Varaverk/unRAID_Essentials/README-Unraid_Essentials.md

# ━━━━━ UNRAID ESSENTIALS ━━━━━

**System-level scripts that act on the unRAID server itself — not containers,
not media, not monitoring.** Keeping the server stable under load, recovering a
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
shutdowns with proper warning sequences.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━

**Server Getting Into Unstable States With No Recovery Path**
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
watchdog tries to restart the container — but Docker itself is barely responding.
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
write log files. The server is functionally dead but still technically running.

Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
threshold breaches. Last line of defense before a hard crash.

**WebGUI Freezing and Nobody Noticing**
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
load, settings don't save, containers can't be started or stopped via the UI. No
container-level alert fires because this isn't a container problem — it's a web
server problem. By the time someone notices it may have been broken for hours.

Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
php-fpm → emhttp. Lightest fix first. Silent when healthy.

**50+ Containers Starting and Filling Syslog With Veth Noise**
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
In the first minute after array start, syslog is buried under 200–400 lines of
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
— a failed mount, a permission error, a service that didn't start — are invisible.

Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.

**WebGUI Queuing Requests Under Load Without Explanation**
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
those workers saturate immediately. New requests queue behind active ones.

Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.

**inotify Exhaustion Producing Unexplained Failures**
When inotify limits are exhausted, containers silently stop receiving filesystem
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
and all containers share the same pool.

Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
run FIRST in ARRAY_START_SCRIPTS before any containers start.

**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
The mover is running — moving a large batch of files from cache to array. A reboot
is triggered. The mover stops mid-file. The file exists partially on both cache and
array simultaneously. unRAID's deduplication layer is confused.

Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
SIGTERM (graceful — finishes current file), SIGKILL only if needed.

---

## ━━━ WHAT THIS FOLDER DOES ━━━

```
WebGUI availability     webgui_watchdog.sh        — nginx → php-fpm → emhttp escalation
Kernel tuning           inotify_tuning.sh        — file watch limits
                        php_fpm_max_children.sh  — PHP worker count
Log hygiene             docker_syslog_filter.sh  — suppress veth noise at start
                        clear_logs.sh            — weekly log trimming
Graceful operations     mover_stop.sh            — clean mover stop
                        rsync_stop.sh            — smart rsync stop (orchestrator-aware)
                        user_scripts_stop.sh     — stop running User Scripts
                        server_reboot.sh         — clean reboot with pre-flight warnings
```

> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.

---

## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

```
Orchestrators/
  array_start.sh ─────────────────────────► inotify_tuning.sh       (first in sequence)
                 ─────────────────────────► docker_syslog_filter.sh  (second)
                 ─────────────────────────► php_fpm_max_children.sh
  weekly_maintenance.sh ──────────────────► clear_logs.sh

  server_reboot.sh ────────────────────────► user_scripts_stop.sh   (called internally)

Watchdogs/
  stability_watchdog.sh and resource_watchdog.sh now live here.
  See Watchdogs/README-Watchdogs.md for how they relate to each other
  and to docker_watchdog.sh and System/storage_watchdog.sh.
```

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | Role | When It Runs |
|--------|------|-------------|
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
Array starts
    │
    ├─ inotify_tuning.sh       ← FIRST — kernel limits inherited at container launch
    ├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
    └─ php_fpm_max_children.sh ← before WebGUI is under load

Every minute (watchdog_orchestrator.sh in Orchestrators/):
    → Watchdogs/resource_watchdog.sh → Watchdogs/docker_watchdog.sh
    → Watchdogs/System/storage_watchdog.sh  → Watchdogs/system_watchdog.sh
    (see Watchdogs/README-Watchdogs.md for full flow)

Every 10 minutes (User Scripts):
    └─ webgui_watchdog.sh
         WebGUI OK → silent exit
         Not responding:
           Step 1: restart nginx       → recheck
           Step 2: restart php-fpm     → recheck
           Step 3: restart emhttp      → recheck
           All failed → notify, exit 1

Weekly (weekly_maintenance.sh):
    └─ clear_logs.sh
         System logs: clear if > LOG_MIN_SIZE_MB
         Docker logs: clear per-container if > LOG_DOCKER_MAX_MB

Manual operations:
    mover_stop.sh        → wall → SIGTERM → SIGKILL → verify stopped
    rsync_stop.sh        → detect orchestrator → kill rsync (or orchestrator+rsync)
    user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
    server_reboot.sh     → pre-flight → wall → wait → VMs → Docker → sync → reboot
```