Varaverk/unRAID_Essentials/README-Unraid_Essentials.md

━━━━━ UNRAID ESSENTIALS ━━━━━

System-level scripts that act on the unRAID server itself — not containers, not media, not monitoring. Keeping the server stable under load, recovering a frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful shutdowns with proper warning sequences.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━

Server Getting Into Unstable States With No Recovery Path A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker watchdog tries to restart the container — but Docker itself is barely responding. The restart hangs. The server needs a reboot, but nothing in the ecosystem is authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The server is functionally dead but still technically running.

Fix: system_watchdog.sh — three-tier response: immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, strike system for sustained threshold breaches. Last line of defense before a hard crash.

WebGUI Freezing and Nobody Noticing The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't load, settings don't save, containers can't be started or stopped via the UI. No container-level alert fires because this isn't a container problem — it's a web server problem. By the time someone notices it may have been broken for hours.

Fix: webgui_restart.sh — checks every 10 minutes, escalates through nginx → php-fpm → emhttp. Lightest fix first. Silent when healthy.

50+ Containers Starting and Filling Syslog With Veth Noise Array starts. 50+ containers come up simultaneously. Docker creates a virtual network interface for each one. Each interface generates multiple syslog entries. In the first minute after array start, syslog is buried under 200–400 lines of veth renamed from eth0 and docker0: port entered forwarding state. Real events — a failed mount, a permission error, a service that didn't start — are invisible.

Fix: docker_syslog_filter.sh — creates an rsyslog drop rule before any container starts. Applied at array start. Idempotent — silent when already correct.

WebGUI Queuing Requests Under Load Without Explanation The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs explains it. The cause: PHP-FPM's pm.max_children defaults to 4–8 workers. With multiple users, active plugins, and 50+ containers potentially hitting the WebGUI, those workers saturate immediately. New requests queue behind active ones.

Fix: php_fpm_max_children.sh — sets pm.max_children=250 at array start. 250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.

inotify Exhaustion Producing Unexplained Failures When inotify limits are exhausted, containers silently stop receiving filesystem events. Arrs don't detect completed downloads. VSCode shows "unable to watch for file changes." Code-Server with node_modules alone can consume 100K–200K watches, and all containers share the same pool.

Fix: inotify_tuning.sh — raises all three inotify limits at array start. Must run FIRST in ARRAY_START_SCRIPTS before any containers start.

Mover Getting Killed Mid-Transfer Leaving Files Inconsistent The mover is running — moving a large batch of files from cache to array. A reboot is triggered. The mover stops mid-file. The file exists partially on both cache and array simultaneously. unRAID's deduplication layer is confused.

Fix: mover_stop.sh — warns users via wall message, waits the configured timeout, SIGTERM (graceful — finishes current file), SIGKILL only if needed.

---

━━━ WHAT THIS FOLDER DOES ━━━

Last-resort stability   system_watchdog.sh       — reboots before crash
Pressure reduction      resource_watchdog.sh     — throttle/pause/stop under load
WebGUI availability     webgui_restart.sh        — nginx → php-fpm → emhttp escalation
Kernel tuning           inotify_tuning.sh        — file watch limits
                        php_fpm_max_children.sh  — PHP worker count
Log hygiene             docker_syslog_filter.sh  — suppress veth noise at start
                        clear_logs.sh            — weekly log trimming
Graceful operations     mover_stop.sh            — clean mover stop
                        rsync_stop.sh            — smart rsync stop (orchestrator-aware)
                        user_scripts_stop.sh     — stop running User Scripts
                        server_reboot.sh         — clean reboot with pre-flight warnings

---

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

Orchestrators/
  array_start.sh ─────────────────────────► inotify_tuning.sh       (first in sequence)
                 ─────────────────────────► docker_syslog_filter.sh  (second)
                 ─────────────────────────► php_fpm_max_children.sh
                 ─────────────────────────► system_watchdog.sh       (background loop)

  watchdog_orchestrator.sh ───────────────► resource_watchdog.sh     (every minute)
  weekly_maintenance.sh ──────────────────► clear_logs.sh

  server_reboot.sh ────────────────────────► user_scripts_stop.sh   (called internally)

Docker_Essentials/
  docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
                                              (mem_shutdown_active flag)

system_watchdog.sh and docker_watchdog.sh (in Docker_Essentials/) are designed to work together — docker_watchdog heals containers first, system_watchdog reboots only when healing has failed. resource_watchdog.sh coordinates with docker_watchdog via the mem_shutdown_active state flag to prevent docker_watchdog from restarting containers that resource_watchdog just stopped to free RAM.

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	Role	When It Runs
system_watchdog.sh	Three-tier last-resort stability watchdog	Continuous background loop via array_start.sh
resource_watchdog.sh	Pressure reduction — throttle/pause/stop under load	Every minute via watchdog_orchestrator.sh
webgui_restart.sh	WebGUI availability — nginx → php-fpm → emhttp	Every 10 min via User Scripts
inotify_tuning.sh	Raise inotify kernel limits	At array start — FIRST
php_fpm_max_children.sh	Set PHP-FPM max worker count	At array start
docker_syslog_filter.sh	Suppress Docker veth syslog noise	At array start — before containers
clear_logs.sh	Size-threshold log cleanup	Weekly via weekly_maintenance.sh
mover_stop.sh	Clean mover stop with SIGTERM → SIGKILL	Manual / before reboot
rsync_stop.sh	Orchestrator-aware rsync stop	Manual
user_scripts_stop.sh	Stop all running User Script processes	Manual / called by server_reboot.sh
server_reboot.sh	Graceful reboot with pre-flight warnings	Manual

---

━━━ HOW THE SCRIPTS RELATE ━━━

Array starts
    │
    ├─ inotify_tuning.sh     ← FIRST — kernel limits inherited at container launch
    ├─ docker_syslog_filter.sh  ← SECOND — before any veth interfaces are created
    ├─ php_fpm_max_children.sh  ← before WebGUI is under load
    └─ system_watchdog.sh    ← starts background loop


Every minute (watchdog_orchestrator.sh):
    └─ resource_watchdog.sh
         Level 1 (soft):   throttle SABnzbd + qBit download speeds
         Level 2 (medium): further throttle + docker pause non-essential containers
         Level 3 (hard):   docker stop optional services + set mem_shutdown_active=true
                           ↓
                     docker_watchdog.sh reads mem_shutdown_active — defers restarts


Every 10 minutes (User Scripts):
    └─ webgui_restart.sh
         WebGUI OK → silent exit
         Not responding:
           Step 1: restart nginx       → recheck
           Step 2: restart php-fpm     → recheck
           Step 3: restart emhttp      → recheck
           All failed → notify, exit 1


Weekly (weekly_maintenance.sh):
    └─ clear_logs.sh
         System logs: clear if > LOG_MIN_SIZE_MB
         Docker logs: clear per-container if > LOG_DOCKER_MAX_MB


Manual operations:
    mover_stop.sh        → wall → SIGTERM → SIGKILL → verify stopped
    rsync_stop.sh        → detect orchestrator → kill rsync (or orchestrator+rsync)
    user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
    server_reboot.sh     → pre-flight → wall → wait → VMs → Docker → sync → reboot