Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites

- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename) - Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API for storage health, service management, mover, user scripts, notifications, disk temps, and platform command validation - Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR, auto-source Plugin/$PLATFORM/adapter.sh after common.sh - Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg, disks.ini, and validate_unraid_cmd calls with platform_*() functions across watchdogs, orchestrators, and System_Essentials scripts - Update all documentation: rename refs, update webgui escalation logic, add platform adapter section to Plugin README, update main README with portability vision and corrected self-healing stack description
2026-06-04 18:14:34 -04:00
parent de50a01ab2
commit 369a9e6c19
73 changed files with 522 additions and 228 deletions
@@ -0,0 +1,556 @@
+# ━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━
+
+Configuration reference, operational procedures, and troubleshooting for
+system-level scripts. Read the ARRAY_START_SCRIPTS order section before
+adding or reordering scripts at array start.
+
+> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
+> now live in `Watchdogs/`. Their configuration reference and troubleshooting
+> procedures are in `Watchdogs/Manual-Watchdogs.md`.
+
+---
+
+## ━━━ CONTENTS ━━━
+
+- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
+- [webgui_watchdog.sh](#webgui_watchdogsh)
+- [inotify_tuning.sh](#inotify_tuningsh)
+- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
+- [docker_syslog_filter.sh](#docker_syslog_filtersh)
+- [clear_logs.sh](#clear_logssh)
+- [mover_stop.sh](#mover_stopsh)
+- [rsync_stop.sh](#rsync_stopsh)
+- [user_scripts_stop.sh](#user_scripts_stopsh)
+- [server_reboot.sh](#server_rebootsh)
+- [Full Configuration Reference](#full-configuration-reference)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Output Tiers
+
+All scripts use a two-tier output model: `echo` lines are always visible; `log`
+lines only appear when `--log` is passed.
+
+**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
+state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
+Per-check detail suppressed.
+
+**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
+`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
+`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
+final summary are visible. Per-item detail suppressed.
+
+---
+
+## ARRAY_START_SCRIPTS Order
+
+> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
+> scripts.** Getting it wrong causes subtle failures that don't show up
+> immediately.
+
+```bash
+# master.conf
+ARRAY_START_SCRIPTS=(
+    "inotify_tuning.sh"           # 1 — FIRST: kernel limits must be set before
+                                   #    any container starts. Containers inherit
+                                   #    inotify limits at launch, not dynamically.
+    "docker_syslog_filter.sh"     # 2 — SECOND: before any veth interfaces are
+                                   #    created. If a container starts first, its
+                                   #    veth creation is already in syslog.
+    "php_fpm_max_children.sh"     # 3 — before WebGUI is under load
+    "ramdisk_setup.sh"            # (from Transcodes/) before Emby starts
+    ...
+)
+# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
+# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
+```
+
+Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
+the old low limits. The limits are kernel-wide — a restart of Code-Server picks
+up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
+
+Why docker_syslog_filter SECOND: The filter must be in place before any container
+starts creating veth interfaces. The first container start after array start
+generates veth messages — these will appear in syslog if the filter isn't active.
+
+---
+
+## webgui_watchdog.sh
+
+### Escalation Logic
+
+```
+curl $WEBGUI_URL → 200 OK → exit 0 (silent)
+
+Not responding:
+  1. platform_restart_service nginx
+     wait WEBGUI_NGINX_WAIT (15s) → recheck
+     → recovered: notify, exit 0
+
+  2. platform_restart_service php-fpm
+     wait WEBGUI_PHP_WAIT (10s) → recheck
+     → recovered: notify, exit 0
+
+  3. platform_restart_service emhttp
+     wait WEBGUI_EMHTTP_WAIT (30s) → recheck
+     → recovered: notify, exit 0
+
+  All three failed → notify warning, exit 1
+```
+
+### Configuration
+
+```bash
+WEBGUI_URL="http://localhost"   # URL to check
+WEBGUI_TIMEOUT=5                # curl timeout in seconds
+WEBGUI_NGINX_WAIT=15            # seconds after nginx restart before recheck
+WEBGUI_PHP_WAIT=10              # seconds after php-fpm restart before recheck
+WEBGUI_EMHTTP_WAIT=30           # seconds after emhttp restart before recheck
+```
+
+### WebGUI Frozen — Manual Recovery
+
+```bash
+# Check which services are running:
+webgui_watchdog.sh --status
+
+# Try manual restart sequence (mirrors what the script calls):
+# Source the ecosystem first to get platform functions:
+source /boot/config/plugins/varaverk/load_config.sh
+platform_restart_service nginx
+# wait 15s, then:
+curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
+
+# If nginx did not fix it, php-fpm:
+platform_restart_service php-fpm
+
+# If still down, emhttp:
+platform_restart_service emhttp
+
+# Raw equivalents (no source needed — paste directly into terminal):
+# /etc/rc.d/rc.nginx restart
+# /etc/rc.d/rc.php-fpm restart
+# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
+
+# If all three failed:
+server_reboot.sh --status   # check for active sessions first
+```
+
+---
+
+## inotify_tuning.sh
+
+### What It Sets
+
+```bash
+INOTIFY_MAX_INSTANCES=1024       # max inotify fd objects per user (default: 128)
+INOTIFY_MAX_WATCHES=1048576      # max watches shared across all users (default: 8192)
+INOTIFY_MAX_QUEUED_EVENTS=32768  # max buffered events (default: 16384)
+```
+
+Verify current values:
+
+```bash
+inotify_tuning.sh --status
+# Shows current vs target for each limit, active instance count, top consumers
+```
+
+### If Code-Server Shows "Unable to Watch for File Changes"
+
+```bash
+# 1. Verify limits are set:
+sysctl fs.inotify.max_user_watches
+# Expected: 1048576
+
+# 2. Check total usage across all containers:
+find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l
+
+# 3. If limits are set but Code-Server still shows the error:
+docker restart Code-Server
+# Running containers inherit limits at launch. Restart picks up the new values.
+
+# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
+inotify_tuning.sh --log
+```
+
+---
+
+## php_fpm_max_children.sh
+
+### What It Sets
+
+```bash
+PHP_MAX_CHILDREN=250             # target pm.max_children (default: 4-8 on unRAID)
+PHP_CONF="/etc/php83/php-fpm.d/www.conf"
+```
+
+250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
+The default of 4–8 saturates immediately under load on a busy server.
+
+### Verify
+
+```bash
+php_fpm_max_children.sh --status
+# Shows current value vs target, PHP-FPM worker count
+
+# Manual verify:
+grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
+# Expected: pm.max_children = 250
+```
+
+### If WebGUI Is Slow Despite the Setting
+
+```bash
+# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
+# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
+# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
+# pm.max_children value may still be too low, or there's a PHP worker leak.
+
+# Check running worker count:
+pgrep -fc php-fpm
+# Compare to pm.max_children — if equal, workers are saturated
+
+# Increase if needed:
+# master.conf: PHP_MAX_CHILDREN=350
+# Then: php_fpm_max_children.sh --log  (will update and restart php-fpm)
+```
+
+---
+
+## docker_syslog_filter.sh
+
+### What It Creates
+
+```
+/etc/rsyslog.d/ignore-docker-veth.conf:
+  if ($msg contains "veth" or $msg contains "docker0") then {
+      stop
+  }
+```
+
+This drops any syslog message containing "veth" or "docker0" before it reaches
+any output target, including the log file. The drop rule is applied at rsyslog
+level — not at the log viewer level.
+
+### Verify
+
+```bash
+docker_syslog_filter.sh --status
+# Shows filter file content and rsyslog process state
+
+# Manual verify:
+cat /etc/rsyslog.d/ignore-docker-veth.conf
+pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"
+
+# Test the filter is active (should produce no syslog output):
+logger "test veth message"
+grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
+```
+
+### If Syslog Still Has Veth Noise
+
+```bash
+# 1. Verify filter file exists with correct content:
+docker_syslog_filter.sh --status
+
+# 2. If content differs — re-apply:
+docker_syslog_filter.sh --log
+
+# 3. Verify rsyslog is using the conf.d directory:
+grep -r "IncludeConfig" /etc/rsyslog.conf
+# Expected: IncludeConfig /etc/rsyslog.d/*.conf  (or similar)
+```
+
+---
+
+## clear_logs.sh
+
+### Thresholds
+
+```bash
+LOG_MIN_SIZE_MB=10       # skip system log if under this — keep recent history
+LOG_DOCKER_MAX_MB=100    # clear Docker container log only if over this
+
+LOG_FILES=(
+    "/var/log/syslog"
+    "/var/log/messages"
+    "/var/log/dmesg"
+)
+```
+
+### Why Truncation Not Deletion
+
+unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
+descriptor open and valid while emptying content — syslogd continues writing to
+the same fd without interruption. Deleting the file would orphan the file
+descriptor and syslog would stop writing until restarted.
+
+### Identifying Large Docker Logs
+
+```bash
+clear_logs.sh --status
+# Shows top 10 Docker logs by size, current size vs threshold
+
+# Find the biggest log manually:
+du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
+```
+
+---
+
+## mover_stop.sh
+
+### Stop Sequence
+
+```
+1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
+2. Wall message to all logged-in terminal users
+3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
+4. SIGTERM — mover finishes its current file operation, then stops
+5. Wait 5 seconds → verify stopped
+6. SIGKILL if still running — forced stop, partial files possible
+7. Final verify — error if still running after SIGKILL
+```
+
+SIGTERM first because the mover can finish the file it is currently moving,
+leaving no partial copies split across cache and array. SIGKILL is a last resort.
+
+### Configuration
+
+```bash
+MOVER_STOP_TIMEOUT=30   # seconds between wall warning and SIGTERM
+```
+
+### Usage
+
+```bash
+mover_stop.sh              # check and stop if running
+mover_stop.sh --status     # show current mover state and PID
+mover_stop.sh --dry-run    # show what would happen without stopping
+```
+
+---
+
+## rsync_stop.sh
+
+### Auto-Detection Logic
+
+rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
+is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.
+
+**Default behavior (orchestrator detected):**
+Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
+next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.
+
+**Default behavior (no orchestrator):**
+Kills rsync directly (standalone rsync.sh run).
+
+**--full-stop:**
+Kills the orchestrator first, then kills rsync. Nothing continues after this.
+Use when everything needs to stop immediately.
+
+### Container Recovery
+
+After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
+for any that were stopped by the interrupted rsync session and restarts them.
+Remote containers are left for docker_watchdog.sh to recover.
+
+Skip container recovery with `--rsync-only` — used when called by other scripts
+that handle recovery themselves.
+
+### Usage
+
+```bash
+rsync_stop.sh                        # smart stop (auto-detect orchestrator)
+rsync_stop.sh --full-stop            # kill orchestrator + rsync
+rsync_stop.sh --rsync-only           # kill rsync, skip container recovery
+rsync_stop.sh --status               # show local and remote rsync state
+rsync_stop.sh --dry-run              # preview without changes
+rsync_stop.sh --full-stop --dry-run  # preview full stop
+```
+
+---
+
+## user_scripts_stop.sh
+
+### Process Identification
+
+Scans `/proc/*/cmdline` for any process whose command line contains
+`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
+`/tmp/user.scripts/` before execution — this signature is reliable regardless of
+what the script is named or how it was launched.
+
+Script names are extracted from the path for display: you see which scripts are
+being stopped, not just PIDs.
+
+### Self-Exclusion
+
+If this script is run via the User Scripts plugin, it would find its own PID in
+the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
+from the kill list.
+
+### Usage
+
+```bash
+user_scripts_stop.sh              # stop all User Script processes
+user_scripts_stop.sh --status     # show running scripts with names and elapsed time
+user_scripts_stop.sh --dry-run    # show what would be stopped
+```
+
+---
+
+## server_reboot.sh
+
+### Full Shutdown Sequence
+
+```
+1. Pre-flight checks (warn only — do not block):
+   - rsync running?     → warn, suggest rsync_stop.sh first
+   - mover running?     → warn, suggest mover_stop.sh first
+   - Emby sessions?     → warn (active streams will be interrupted)
+
+2. Wall message to all logged-in terminal users
+
+3. unRAID dashboard notification
+
+4. Wait REBOOT_SLEEP seconds (default: 30)
+
+5. Graceful VM shutdown:
+   virsh shutdown <each VM>  (ACPI signal — clean shutdown)
+   Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond
+
+6. /etc/rc.d/rc.libvirt stop   (VM Manager)
+
+7. Orchestrators/array_stopping.sh — safe ordered array stop:
+   - user_scripts_stop.sh        stop background User Scripts
+   - fallback.sh --stop          graceful fallback teardown
+   - rsync_stop.sh --rsync-only  kill active rsync transfers
+   - mover_stop.sh               stop mover
+   - docker_container_stop.sh    stop all containers gracefully
+
+8. sync   (flush filesystem buffers to disk)
+
+9. /sbin/reboot
+```
+
+### Configuration
+
+```bash
+REBOOT_SLEEP=30      # seconds between warning and shutdown sequence
+REBOOT_VM_WAIT=30    # seconds to wait for VMs to shut down gracefully
+```
+
+### Recommended Pre-Reboot Sequence
+
+For a clean reboot when services are active:
+
+```bash
+rsync_stop.sh          # stop any active rsync (smart mode)
+mover_stop.sh          # stop mover gracefully
+server_reboot.sh --status  # check what's still running
+server_reboot.sh --reason="planned maintenance"
+```
+
+### Usage
+
+```bash
+server_reboot.sh                         # reboot with 30s warning
+server_reboot.sh --dry-run               # walk through without rebooting
+server_reboot.sh --status                # show what would be affected
+server_reboot.sh --reason="disk work"    # include reason in notification
+```
+
+---
+
+## Full Configuration Reference
+
+> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
+> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
+
+```bash
+# master.conf
+
+# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
+WEBGUI_URL="http://localhost"
+WEBGUI_TIMEOUT=5
+WEBGUI_NGINX_WAIT=15
+WEBGUI_PHP_WAIT=10
+WEBGUI_EMHTTP_WAIT=30
+
+# ── inotify Tuning ─────────────────────────────────────────────────────────────
+INOTIFY_MAX_INSTANCES=1024
+INOTIFY_MAX_WATCHES=1048576
+INOTIFY_MAX_QUEUED_EVENTS=32768
+
+# ── PHP-FPM ────────────────────────────────────────────────────────────────────
+PHP_MAX_CHILDREN=250
+PHP_CONF="/etc/php83/php-fpm.d/www.conf"
+
+# ── Syslog Filter ──────────────────────────────────────────────────────────────
+FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
+
+# ── Log Cleaner ────────────────────────────────────────────────────────────────
+LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
+LOG_MIN_SIZE_MB=10
+LOG_DOCKER_MAX_MB=100
+
+# ── Mover Stop ─────────────────────────────────────────────────────────────────
+MOVER_STOP_TIMEOUT=30
+
+# ── Server Reboot ──────────────────────────────────────────────────────────────
+REBOOT_SLEEP=30
+REBOOT_VM_WAIT=30
+```
+
+---
+
+## Troubleshooting
+
+> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
+> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
+
+### rsync_stop Killed the Wrong Thing
+
+If --full-stop killed an orchestrator you didn't intend to kill:
+
+```bash
+# Next time use default mode (no --full-stop) to kill only rsync subprocess.
+# To verify what would be killed before running:
+rsync_stop.sh --status       # shows running rsync and detected orchestrators
+rsync_stop.sh --dry-run      # shows smart mode decision
+rsync_stop.sh --full-stop --dry-run  # shows full-stop decision
+```
+
+### WebGUI Recovery After All Three Steps Failed
+
+```bash
+# Check if processes are running:
+pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
+pgrep emhttpd   && echo "emhttp: yes" || echo "emhttp: no"
+pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
+
+# Check recent nginx errors:
+cat /var/log/nginx/error.log | tail -20
+
+# Check emhttp log:
+tail -20 /var/log/syslog | grep emhttp
+
+# Last resort — reboot:
+server_reboot.sh --reason="WebGUI unrecoverable"
+```
+
+### PHP-FPM Config Not Found After unRAID Update
+
+unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
+errors with "config file not found":
+
+```bash
+# Find the new config path:
+find /etc -name "www.conf" 2>/dev/null
+
+# Update PHP_CONF in master.conf:
+PHP_CONF="/etc/php84/php-fpm.d/www.conf"   # example for php84
+
+# Verify with:
+php_fpm_max_children.sh --status
+```