Structural reorganization: watchdog taxonomy + server_reboot integration

Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown

unRAID_Essentials/Manual-Unraid_Essentials.md

@@ -13,7 +13,7 @@ adding or reordering scripts at array start.
 ## ━━━ CONTENTS ━━━
 
 - [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
-- [webgui_restart.sh](#webgui_restartsh)
+- [webgui_watchdog.sh](#webgui_watchdogsh)
 - [inotify_tuning.sh](#inotify_tuningsh)
 - [php_fpm_max_children.sh](#php_fpm_max_childrensh)
 - [docker_syslog_filter.sh](#docker_syslog_filtersh)
@@ -32,7 +32,7 @@ adding or reordering scripts at array start.
 All scripts use a two-tier output model: `echo` lines are always visible; `log`
 lines only appear when `--log` is passed.
 
-**Daemon scripts** (`webgui_restart.sh`): run on every cycle. Without `--log`, only
+**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
 state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
 Per-check detail suppressed.
 
@@ -76,7 +76,7 @@ generates veth messages — these will appear in syslog if the filter isn't acti
 
 ---
 
-## webgui_restart.sh
+## webgui_watchdog.sh
 
 ### Escalation Logic
 
@@ -113,7 +113,7 @@ WEBGUI_EMHTTP_WAIT=30           # seconds after emhttp restart before recheck
 
 ```bash
 # Check which services are running:
-webgui_restart.sh --status
+webgui_watchdog.sh --status
 
 # Try manual restart sequence (same as the script):
 /etc/rc.d/rc.nginx restart
@@ -196,8 +196,8 @@ grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
 
 ```bash
 # Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
-# Look at the webgui_restart.sh escalation — step 2 (php-fpm restart) is specifically
-# for worker exhaustion. If webgui_restart.sh is regularly hitting step 2, the
+# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
+# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
 # pm.max_children value may still be too low, or there's a PHP worker leak.
 
 # Check running worker count:

unRAID_Essentials/README-Unraid_Essentials.md

@@ -27,7 +27,7 @@ load, settings don't save, containers can't be started or stopped via the UI. No
 container-level alert fires because this isn't a container problem — it's a web
 server problem. By the time someone notices it may have been broken for hours.
 
-Fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
+Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
 php-fpm → emhttp. Lightest fix first. Silent when healthy.
 
 **50+ Containers Starting and Filling Syslog With Veth Noise**
@@ -71,7 +71,7 @@ SIGTERM (graceful — finishes current file), SIGKILL only if needed.
 ## ━━━ WHAT THIS FOLDER DOES ━━━
 
 ```
-WebGUI availability     webgui_restart.sh        — nginx → php-fpm → emhttp escalation
+WebGUI availability     webgui_watchdog.sh        — nginx → php-fpm → emhttp escalation
 Kernel tuning           inotify_tuning.sh        — file watch limits
                         php_fpm_max_children.sh  — PHP worker count
 Log hygiene             docker_syslog_filter.sh  — suppress veth noise at start
@@ -110,7 +110,7 @@ Watchdogs/
 
 | Script | Role | When It Runs |
 |--------|------|-------------|
-| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
+| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
 | `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
 | `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
 | `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
@@ -133,11 +133,11 @@ Array starts
 
 Every minute (watchdog_orchestrator.sh in Orchestrators/):
     → Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
-    → Watchdogs/storage_watchdog.sh  → Watchdogs/system_watchdog.sh
+    → Watchdogs/System/storage_watchdog.sh  → Watchdogs/system_watchdog.sh
     (see Watchdogs/README-Watchdogs.md for full flow)
 
 Every 10 minutes (User Scripts):
-    └─ webgui_restart.sh
+    └─ webgui_watchdog.sh
          WebGUI OK → silent exit
          Not responding:
            Step 1: restart nginx       → recheck

unRAID_Essentials/server_reboot.sh

@@ -19,9 +19,9 @@
 #   2. Wall message to all logged-in terminal users
 #   3. unRAID dashboard notification
 #   4. Wait REBOOT_SLEEP seconds — users time to save work
-#   5. Graceful VM shutdown via virsh — ACPI signal, then wait REBOOT_VM_WAIT
-#   6. Stop libvirt (VM Manager)
-#   7. Stop Docker service
+#   5. array_stopping.sh — user scripts, rsync, mover, containers (verified stop)
+#   6. Graceful VM shutdown via virsh — ACPI signal, then wait REBOOT_VM_WAIT
+#   7. Stop libvirt (VM Manager)
 #   8. sync — filesystem buffers flushed to disk
 #   9. /sbin/reboot
 #
@@ -121,15 +121,6 @@ acquire_lock
 
 detect_hosts
 
-# ── Exit Trap — restart Docker service if reboot sequence aborts after stopping it ────────────
-_REBOOT_DOCKER_STOPPED=false
-_trap_restart_docker_service() {
-    [[ "$_REBOOT_DOCKER_STOPPED" == true ]] || return
-    warn "Exit trap: restarting Docker service after aborted reboot sequence"
-    /etc/rc.d/rc.docker start >/dev/null 2>&1 || true
-}
-trap _trap_restart_docker_service EXIT
-
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made, no reboot will occur"
 
 # ==============================================================================================
@@ -239,6 +230,25 @@ if [[ "$REBOOT_SLEEP" -gt 0 ]]; then
     fi
 fi
 
+# ==============================================================================================
+# ━━━ Array Stop Orchestrator ━━━
+# ==============================================================================================
+echo ""
+echo "━━━ $ICON_STOP Array Stop Orchestrator ━━━"
+ARRAY_STOP_SCRIPT="$SCRIPT_DIR/../Orchestrators/array_stopping.sh"
+
+if [[ ! -f "$ARRAY_STOP_SCRIPT" ]]; then
+    warn "array_stopping.sh not found — skipping orchestrated stop"
+elif [[ "$DRY_RUN" == true ]]; then
+    bash "$ARRAY_STOP_SCRIPT" --dry-run
+else
+    if bash "$ARRAY_STOP_SCRIPT"; then
+        log "Array stop complete ✅"
+    else
+        warn "array_stopping.sh reported failures — proceeding with reboot"
+    fi
+fi
+
 # ==============================================================================================
 # ━━━ Graceful VM Shutdown ━━━
 # ==============================================================================================
@@ -280,22 +290,6 @@ else
     fi
 fi
 
-# ==============================================================================================
-# ━━━ Stop Docker ━━━
-# ==============================================================================================
-echo ""
-echo "━━━ $ICON_CONTAINERS Stop Docker ━━━"
-if [[ "$DRY_RUN" == true ]]; then
-    warn "DRY RUN — would stop Docker service"
-else
-    if /etc/rc.d/rc.docker stop >/dev/null 2>&1; then
-        _REBOOT_DOCKER_STOPPED=true
-        warn "Docker stopped ✅"
-    else
-        warn "Docker stop returned non-zero — may already be stopped"
-    fi
-fi
-
 # ==============================================================================================
 # ━━━ Sync Disks ━━━
 # ==============================================================================================
@@ -327,6 +321,5 @@ if [[ "$DRY_RUN" == true ]]; then
 else
     warn "$ICON_REBOOT Rebooting $MY_ID now..."
     echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-    trap - EXIT  # committed to reboot — Docker should stay down
     /sbin/reboot
 fi

unRAID_Essentials/webgui_restart.sh

@@ -1,302 +0,0 @@
-#!/bin/bash
-# ==============================================================================================
-# ================================= WebGUI Watchdog ============================================
-# ==============================================================================================
-#
-# PURPOSE
-# ─────────────────────────────────────────────────────────────────────────────
-# Monitors the unRAID WebGUI and restarts services if unresponsive. Uses a
-# three-step escalating strategy — lightest fix first, heaviest last. Run
-# every 5–10 minutes via the User Scripts plugin. Silent when healthy.
-#
-# ==============================================================================================
-# OPERATIONAL MODEL
-# ==============================================================================================
-#
-# Escalation Path
-#   WebGUI responding → log() + exit 0 (completely silent ✅)
-#
-#   Not responding:
-#     Step 1 — nginx restart
-#               Lightest fix — handles most transient WebGUI failures:
-#               nginx crash, worker stuck, connection timeout.
-#               Wait WEBGUI_NGINX_WAIT seconds → recheck.
-#
-#     Step 2 — php-fpm restart
-#               WebGUI runs through PHP-FPM. Worker exhaustion causes silent
-#               failure — requests queue and the WebGUI appears frozen.
-#               Wait WEBGUI_PHP_WAIT seconds → recheck.
-#
-#     Step 3 — emhttp restart
-#               Heaviest fix. emhttp is the unRAID management daemon.
-#               Array, Docker, and shares stay running — only WebGUI
-#               management restarts. Takes longer — WEBGUI_EMHTTP_WAIT.
-#               Wait WEBGUI_EMHTTP_WAIT seconds → recheck.
-#
-#     All three failed → notify, manual intervention needed → exit 1.
-#
-# ==============================================================================================
-# OPERATIONAL SAFEGUARDS
-# ==============================================================================================
-#
-# Root Required
-#   Service restart commands require root.
-#
-# Single Instance Lock
-#   acquire_lock prevents concurrent runs double-restarting services.
-#
-# Process Verify After Each Restart
-#   pgrep check after each rc.* command — errors if process not running.
-#
-# Silent When Healthy
-#   Completely silent on healthy cycles. Only produces output when recovering.
-#
-# ==============================================================================================
-# CONFIGURATION
-# ==============================================================================================
-#
-# master.conf
-#
-#   WEBGUI_URL
-#       URL to check for WebGUI response. (default: http://localhost)
-#
-#   WEBGUI_TIMEOUT
-#       curl timeout in seconds. (default: 5)
-#
-#   WEBGUI_NGINX_WAIT
-#       Seconds after nginx restart before rechecking. (default: 15)
-#
-#   WEBGUI_PHP_WAIT
-#       Seconds after php-fpm restart before rechecking. (default: 10)
-#
-#   WEBGUI_EMHTTP_WAIT
-#       Seconds after emhttp restart before rechecking. (default: 30)
-#
-# ==============================================================================================
-# RUNTIME MODES
-# ==============================================================================================
-#
-# webgui_restart.sh
-#     Check WebGUI. Escalate through nginx → php-fpm → emhttp if unresponsive.
-#
-# webgui_restart.sh --dry-run
-#     Show which services would be restarted. No restarts, no waits.
-#
-# webgui_restart.sh --status
-#     Show current WebGUI response state and nginx/php-fpm/emhttp process states.
-#
-# webgui_restart.sh --log
-#     Verbose output — show each check, each restart attempt, each wait.
-#
-# ==============================================================================================
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-
-source "$SCRIPT_DIR/../load_config.sh"
-
-parse_args "$@"
-
-# ==============================================================================================
-# ━━━ Setup ━━━
-# ==============================================================================================
-if [[ "$EUID" -ne 0 ]]; then
-    error "Must be run as root"
-    exit 1
-fi
-
-validate_unraid_cmd \
-    "/usr/local/emhttp/plugins/dynamix/scripts/notify" \
-    "" "" \
-    "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
-
-acquire_lock
-
-detect_hosts
-
-[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no services will be restarted"
-
-# ==============================================================================================
-# ━━━ Status ━━━
-# ==============================================================================================
-if [[ "$SHOW_STATUS" == true ]]; then
-    echo ""
-    echo "━━━━━ $ICON_SUMMARY WEBGUI WATCHDOG STATUS ━━━━━"
-    echo "$ICON_HOST    Identity:    $MY_ID ($LOCAL_SERVER_NAME)"
-    echo "$ICON_WEBGUI  URL:         $WEBGUI_URL"
-    echo "$ICON_WEBGUI  Timeouts:    curl=${WEBGUI_TIMEOUT}s nginx=${WEBGUI_NGINX_WAIT}s php=${WEBGUI_PHP_WAIT:-10}s emhttp=${WEBGUI_EMHTTP_WAIT}s"
-    echo ""
-
-    if curl -sf --max-time "$WEBGUI_TIMEOUT" "$WEBGUI_URL" >/dev/null 2>&1; then
-        echo "  $ICON_SUCCESS WebGUI:    responding ✅"
-    else
-        echo "  $ICON_ERROR  WebGUI:    NOT responding"
-    fi
-
-    pgrep -x nginx >/dev/null 2>&1 && \
-        echo "  $ICON_SUCCESS nginx:     running ✅" || \
-        echo "  $ICON_ERROR  nginx:     NOT running"
-
-    pgrep -f "php-fpm" >/dev/null 2>&1 && \
-        FPM_COUNT=$(pgrep -fc "php-fpm" 2>/dev/null || echo "?") && \
-        echo "  $ICON_SUCCESS php-fpm:   running ($FPM_COUNT workers) ✅" || \
-        echo "  $ICON_ERROR  php-fpm:   NOT running"
-
-    pgrep emhttpd >/dev/null 2>&1 && \
-        echo "  $ICON_SUCCESS emhttp:    running ✅" || \
-        echo "  $ICON_ERROR  emhttp:    NOT running"
-    echo "━━━━━━━━━━━━━━━━━━━━━━━"
-    exit 0
-fi
-
-# ==============================================================================================
-# ── CHECK AND ESCALATE ────────────────────────────────────────────────────────────────────────
-# ==============================================================================================
-check_webgui() {
-    curl -sf --max-time "$WEBGUI_TIMEOUT" "$WEBGUI_URL" >/dev/null 2>&1
-}
-
-START=$(date +%s)
-RECOVERY_ACTION=""
-RECOVERY_OK=false
-
-log "WebGUI check — $WEBGUI_URL"
-
-# ── Healthy — completely silent ───────────────────────────────────────────────────────────────
-if check_webgui; then
-    log "WebGUI responding — healthy ✅"
-    exit 0
-fi
-
-# ── Not responding — begin escalation ────────────────────────────────────────────────────────
-echo ""
-echo "━━━ $ICON_WEBGUI WebGUI Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
-echo "$ICON_HOST $MY_ID ($LOCAL_SERVER_NAME)"
-echo ""
-warn "WebGUI not responding at $WEBGUI_URL — beginning escalation"
-
-# ── Step 1 — nginx restart ────────────────────────────────────────────────────────────────────
-echo ""
-echo "━━━ Step 1 — nginx Restart ━━━"
-
-if [[ "$DRY_RUN" == true ]]; then
-    warn "DRY RUN — would restart nginx"
-else
-    warn "Restarting nginx..."
-    if /etc/rc.d/rc.nginx restart >/dev/null 2>&1; then
-        # Verify nginx actually running
-        sleep 2
-        if pgrep -x nginx >/dev/null 2>&1; then
-            warn "nginx restarted ✅"
-        else
-            error "nginx not running after restart command"
-        fi
-    else
-        error "nginx restart command failed"
-    fi
-
-    log "Waiting ${WEBGUI_NGINX_WAIT}s for nginx to recover..."
-    sleep "$WEBGUI_NGINX_WAIT"
-
-    if check_webgui; then
-        RECOVERY_ACTION="nginx restart"
-        RECOVERY_OK=true
-    fi
-fi
-
-# ── Step 2 — php-fpm restart ──────────────────────────────────────────────────────────────────
-if [[ "$RECOVERY_OK" == false ]]; then
-    echo ""
-    echo "━━━ Step 2 — php-fpm Restart ━━━"
-    warn "WebGUI still not responding — restarting php-fpm"
-    warn "WebGUI may be frozen due to worker exhaustion (check system_tuning_monitor.sh)"
-
-    if [[ "$DRY_RUN" == true ]]; then
-        warn "DRY RUN — would restart php-fpm"
-    else
-        if /etc/rc.d/rc.php-fpm restart >/dev/null 2>&1; then
-            sleep 2
-            if pgrep -f "php-fpm" >/dev/null 2>&1; then
-                warn "php-fpm restarted ✅"
-            else
-                error "php-fpm not running after restart command"
-            fi
-        else
-            error "php-fpm restart command failed"
-        fi
-
-        log "Waiting ${WEBGUI_PHP_WAIT:-10}s for php-fpm to recover..."
-        sleep "${WEBGUI_PHP_WAIT:-10}"
-
-        if check_webgui; then
-            RECOVERY_ACTION="php-fpm restart"
-            RECOVERY_OK=true
-        fi
-    fi
-fi
-
-# ── Step 3 — emhttp restart ───────────────────────────────────────────────────────────────────
-if [[ "$RECOVERY_OK" == false ]]; then
-    echo ""
-    echo "━━━ Step 3 — emhttp Restart ━━━"
-    warn "WebGUI still not responding — restarting emhttp (unRAID management daemon)"
-    warn "Array, Docker, and shares remain running — WebGUI management will briefly restart"
-
-    if [[ "$DRY_RUN" == true ]]; then
-        warn "DRY RUN — would restart emhttp"
-    else
-        if /usr/local/sbin/emhttp stop >/dev/null 2>&1 && /usr/local/sbin/emhttp start >/dev/null 2>&1; then
-            sleep 2
-            if pgrep emhttpd >/dev/null 2>&1; then
-                warn "emhttp restarted ✅"
-            else
-                error "emhttp not running after restart command"
-            fi
-        else
-            error "emhttp restart command failed"
-        fi
-
-        log "Waiting ${WEBGUI_EMHTTP_WAIT}s for emhttp to recover..."
-        sleep "$WEBGUI_EMHTTP_WAIT"
-
-        if check_webgui; then
-            RECOVERY_ACTION="emhttp restart"
-            RECOVERY_OK=true
-        fi
-    fi
-fi
-
-END=$(date +%s)
-
-# ==============================================================================================
-# ━━━ Summary ━━━
-# ==============================================================================================
-echo ""
-echo "━━━━━ $ICON_SUMMARY WEBGUI WATCHDOG SUMMARY ━━━━━"
-echo "$ICON_HOST    Identity: $MY_ID ($LOCAL_SERVER_NAME)"
-echo "$ICON_WEBGUI  URL:      $WEBGUI_URL"
-echo "$ICON_TIME    Duration: $(format_duration $(( END - START )))"
-echo ""
-
-if [[ "$DRY_RUN" == true ]]; then
-    warn "DRY RUN — no services restarted"
-    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-    exit 0
-fi
-
-if [[ "$RECOVERY_OK" == true ]]; then
-    warn "$ICON_SUCCESS WebGUI recovered via: $RECOVERY_ACTION"
-    notify "WebGUI recovered on $(hostname) ($MY_ID) via $RECOVERY_ACTION — monitor for recurrence" \
-        "WebGUI Watchdog" "warning"
-else
-    echo "$ICON_ERROR Status: UNRECOVERED — all three restart steps failed"
-    echo "$ICON_ERROR Manual intervention needed:"
-    echo "    1. Check: pgrep nginx; pgrep emhttpd"
-    echo "    2. Check: journalctl -u nginx --since '10 minutes ago'"
-    echo "    3. Try: server_reboot.sh if nothing else works"
-    notify "WebGUI UNRECOVERED on $(hostname) ($MY_ID) — nginx + php-fpm + emhttp restart all failed — manual intervention needed" \
-        "WebGUI Watchdog" "warning"
-fi
-echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
-
-[[ "$RECOVERY_OK" == false && "$DRY_RUN" == false ]] && exit 1
-exit 0