Varaverk/System_Essentials/README-System_Essentials.md

━━━━━ SYSTEM ESSENTIALS ━━━━━

System-level scripts that act on the server itself — not containers, not media, not monitoring. Keeping the server stable under load, recovering a frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful shutdowns with proper warning sequences.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━

Server Getting Into Unstable States With No Recovery Path A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker watchdog tries to restart the container — but Docker itself is barely responding. The restart hangs. The server needs a reboot, but nothing in the ecosystem is authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The server is functionally dead but still technically running.

Fix: stability_watchdog.sh — three-tier response: immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, strike system for sustained threshold breaches. Last line of defense before a hard crash.

WebGUI Freezing and Nobody Noticing The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't load, settings don't save, containers can't be started or stopped via the UI. No container-level alert fires because this isn't a container problem — it's a web server problem. By the time someone notices it may have been broken for hours.

Fix: webgui_watchdog.sh — checks every 10 minutes, escalates through nginx → php-fpm → emhttp. Lightest fix first. Silent when healthy.

50+ Containers Starting and Filling Syslog With Veth Noise Array starts. 50+ containers come up simultaneously. Docker creates a virtual network interface for each one. Each interface generates multiple syslog entries. In the first minute after array start, syslog is buried under 200–400 lines of veth renamed from eth0 and docker0: port entered forwarding state. Real events — a failed mount, a permission error, a service that didn't start — are invisible.

Fix: docker_syslog_filter.sh — creates an rsyslog drop rule before any container starts. Applied at array start. Idempotent — silent when already correct.

WebGUI Queuing Requests Under Load Without Explanation The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs explains it. The cause: PHP-FPM's pm.max_children defaults to 4–8 workers. With multiple users, active plugins, and 50+ containers potentially hitting the WebGUI, those workers saturate immediately. New requests queue behind active ones.

Fix: php_fpm_max_children.sh — sets pm.max_children=250 at array start. 250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.

inotify Exhaustion Producing Unexplained Failures When inotify limits are exhausted, containers silently stop receiving filesystem events. Arrs don't detect completed downloads. VSCode shows "unable to watch for file changes." Code-Server with node_modules alone can consume 100K–200K watches, and all containers share the same pool.

Fix: inotify_tuning.sh — raises all three inotify limits at array start. Must run FIRST in ARRAY_START_SCRIPTS before any containers start.

Mover Getting Killed Mid-Transfer Leaving Files Inconsistent The mover is running — moving a large batch of files from cache to array. A reboot is triggered. The mover stops mid-file. The file exists partially on both cache and array simultaneously. unRAID's deduplication layer is confused.

Fix: mover_stop.sh — warns users via wall message, waits the configured timeout, SIGTERM (graceful — finishes current file), SIGKILL only if needed.

---

━━━ WHAT THIS FOLDER DOES ━━━

WebGUI availability     webgui_watchdog.sh        — nginx → php-fpm → emhttp escalation
Kernel tuning           inotify_tuning.sh        — file watch limits
                        php_fpm_max_children.sh  — PHP worker count
Log hygiene             docker_syslog_filter.sh  — suppress veth noise at start
                        clear_logs.sh            — weekly log trimming
Graceful operations     mover_stop.sh            — clean mover stop
                        rsync_stop.sh            — smart rsync stop (orchestrator-aware)
                        user_scripts_stop.sh     — stop running User Scripts
                        server_reboot.sh         — clean reboot with pre-flight warnings

stability_watchdog.sh and resource_watchdog.sh have moved to Watchdogs/. See Watchdogs/README-Watchdogs.md for the full watchdog suite.

---

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

Orchestrators/
  array_started.sh ─────────────────────────► inotify_tuning.sh       (first in sequence)
                 ─────────────────────────► docker_syslog_filter.sh  (second)
                 ─────────────────────────► php_fpm_max_children.sh
  weekly_maintenance.sh ──────────────────► clear_logs.sh

  server_reboot.sh ────────────────────────► user_scripts_stop.sh   (called internally)

Watchdogs/
  stability_watchdog.sh and resource_watchdog.sh now live here.
  See Watchdogs/README-Watchdogs.md for how they relate to each other
  and to docker_watchdog.sh and System/storage_watchdog.sh.

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	Role	When It Runs
webgui_watchdog.sh	WebGUI availability — nginx → php-fpm → emhttp	Every minute via watchdog_orchestrator → system_watchdog
inotify_tuning.sh	Raise inotify kernel limits	At array start — FIRST
php_fpm_max_children.sh	Set PHP-FPM max worker count	At array start
docker_syslog_filter.sh	Suppress Docker veth syslog noise	At array start — before containers
clear_logs.sh	Size-threshold log cleanup	Weekly via weekly_maintenance.sh
mover_stop.sh	Clean mover stop with SIGTERM → SIGKILL	Manual / before reboot
rsync_stop.sh	Orchestrator-aware rsync stop	Manual
user_scripts_stop.sh	Stop all running User Script processes	Manual / called by server_reboot.sh
server_reboot.sh	Graceful reboot with pre-flight warnings	Manual

---

━━━ HOW THE SCRIPTS RELATE ━━━

Array starts
    │
    ├─ inotify_tuning.sh       ← FIRST — kernel limits inherited at container launch
    ├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
    └─ php_fpm_max_children.sh ← before WebGUI is under load

Every minute (watchdog_orchestrator.sh in Orchestrators/):
    → Watchdogs/resource_watchdog.sh
    → Watchdogs/docker_watchdog.sh
    → Watchdogs/system_watchdog.sh (thin orchestrator)
        └─ Watchdogs/System/storage_watchdog.sh
        └─ Watchdogs/System/webgui_watchdog.sh
               WebGUI OK → silent exit
               Not responding:
                 Step 1: restart nginx       → recheck
                 Step 2: restart php-fpm     → recheck
                 Step 3: restart emhttp      → recheck
                 All failed → notify, exit 1
        └─ Watchdogs/System/network_watchdog.sh
    → Watchdogs/stability_watchdog.sh
    (see Watchdogs/README-Watchdogs.md for full flow)

Weekly (weekly_maintenance.sh):
    └─ clear_logs.sh
         System logs: clear if > LOG_MIN_SIZE_MB
         Docker logs: clear per-container if > LOG_DOCKER_MAX_MB

Manual operations:
    mover_stop.sh        → wall → SIGTERM → SIGKILL → verify stopped
    rsync_stop.sh        → detect orchestrator → kill rsync (or orchestrator+rsync)
    user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
    server_reboot.sh     → pre-flight → wall → wait → VMs → Docker → sync → reboot