Structural reorganization: watchdog taxonomy + server_reboot integration

Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
This commit is contained in:
Gmer4Lfe
2026-05-22 20:17:22 -04:00
parent 670f25fbbd
commit cf21efaeea
18 changed files with 962 additions and 824 deletions
@@ -27,7 +27,7 @@ load, settings don't save, containers can't be started or stopped via the UI. No
container-level alert fires because this isn't a container problem — it's a web
server problem. By the time someone notices it may have been broken for hours.
Fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
php-fpm → emhttp. Lightest fix first. Silent when healthy.
**50+ Containers Starting and Filling Syslog With Veth Noise**
@@ -71,7 +71,7 @@ SIGTERM (graceful — finishes current file), SIGKILL only if needed.
## ━━━ WHAT THIS FOLDER DOES ━━━
```
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
Kernel tuning inotify_tuning.sh — file watch limits
php_fpm_max_children.sh — PHP worker count
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
@@ -110,7 +110,7 @@ Watchdogs/
| Script | Role | When It Runs |
|--------|------|-------------|
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
@@ -133,11 +133,11 @@ Array starts
Every minute (watchdog_orchestrator.sh in Orchestrators/):
→ Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
→ Watchdogs/storage_watchdog.sh → Watchdogs/system_watchdog.sh
→ Watchdogs/System/storage_watchdog.sh → Watchdogs/system_watchdog.sh
(see Watchdogs/README-Watchdogs.md for full flow)
Every 10 minutes (User Scripts):
└─ webgui_restart.sh
└─ webgui_watchdog.sh
WebGUI OK → silent exit
Not responding:
Step 1: restart nginx → recheck