Gmer4Lfe
070013cb4f
array_start.sh → array_started.sh (bulk rename — actual file has always been array_started.sh; docs consistently used the wrong name across 9 files) failover.sh → fallback.sh (bulk rename — script is Fallback/fallback.sh; docs used the old name throughout) FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1 → FALLBACK_HOST2_COVERS_HOST1_TIER1 (README.md — wrong variable name, actual vars use FALLBACK_ prefix and COVERS_ not RUNS_FOR_) docker_watchdog.sh attribution: "started by array_started.sh" → "called every minute by watchdog_orchestrator.sh" (docker_watchdog is not in ARRAY_START_SCRIPTS) server_reboot.sh shutdown sequence: step 7 "rc.docker stop" → calls array_stopping.sh (user_scripts, fallback, rsync, mover, docker_container_stop) Rsync manual ARRAY_START_SCRIPTS list: removed stability_watchdog and docker_watchdog (now in watchdog_orchestrator); failover → fallback
161 lines
8.0 KiB
Markdown
161 lines
8.0 KiB
Markdown
# ━━━━━ UNRAID ESSENTIALS ━━━━━
|
||
|
||
**System-level scripts that act on the unRAID server itself — not containers,
|
||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||
shutdowns with proper warning sequences.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||
|
||
**Server Getting Into Unstable States With No Recovery Path**
|
||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||
write log files. The server is functionally dead but still technically running.
|
||
|
||
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
|
||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||
threshold breaches. Last line of defense before a hard crash.
|
||
|
||
**WebGUI Freezing and Nobody Noticing**
|
||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||
container-level alert fires because this isn't a container problem — it's a web
|
||
server problem. By the time someone notices it may have been broken for hours.
|
||
|
||
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
|
||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||
|
||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||
network interface for each one. Each interface generates multiple syslog entries.
|
||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||
|
||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||
starts. Applied at array start. Idempotent — silent when already correct.
|
||
|
||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||
those workers saturate immediately. New requests queue behind active ones.
|
||
|
||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||
|
||
**inotify Exhaustion Producing Unexplained Failures**
|
||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||
and all containers share the same pool.
|
||
|
||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||
|
||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||
array simultaneously. unRAID's deduplication layer is confused.
|
||
|
||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||
|
||
```
|
||
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
|
||
Kernel tuning inotify_tuning.sh — file watch limits
|
||
php_fpm_max_children.sh — PHP worker count
|
||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||
clear_logs.sh — weekly log trimming
|
||
Graceful operations mover_stop.sh — clean mover stop
|
||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||
user_scripts_stop.sh — stop running User Scripts
|
||
server_reboot.sh — clean reboot with pre-flight warnings
|
||
```
|
||
|
||
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
|
||
|
||
---
|
||
|
||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||
|
||
```
|
||
Orchestrators/
|
||
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||
─────────────────────────► docker_syslog_filter.sh (second)
|
||
─────────────────────────► php_fpm_max_children.sh
|
||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||
|
||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||
|
||
Watchdogs/
|
||
stability_watchdog.sh and resource_watchdog.sh now live here.
|
||
See Watchdogs/README-Watchdogs.md for how they relate to each other
|
||
and to docker_watchdog.sh and System/storage_watchdog.sh.
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||
|
||
| Script | Role | When It Runs |
|
||
|--------|------|-------------|
|
||
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
|
||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||
|
||
---
|
||
|
||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||
|
||
```
|
||
Array starts
|
||
│
|
||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||
└─ php_fpm_max_children.sh ← before WebGUI is under load
|
||
|
||
Every minute (watchdog_orchestrator.sh in Orchestrators/):
|
||
→ Watchdogs/resource_watchdog.sh
|
||
→ Watchdogs/docker_watchdog.sh
|
||
→ Watchdogs/system_watchdog.sh (thin orchestrator)
|
||
└─ Watchdogs/System/storage_watchdog.sh
|
||
└─ Watchdogs/System/webgui_watchdog.sh
|
||
WebGUI OK → silent exit
|
||
Not responding:
|
||
Step 1: restart nginx → recheck
|
||
Step 2: restart php-fpm → recheck
|
||
Step 3: restart emhttp → recheck
|
||
All failed → notify, exit 1
|
||
└─ Watchdogs/System/network_watchdog.sh
|
||
→ Watchdogs/stability_watchdog.sh
|
||
(see Watchdogs/README-Watchdogs.md for full flow)
|
||
|
||
Weekly (weekly_maintenance.sh):
|
||
└─ clear_logs.sh
|
||
System logs: clear if > LOG_MIN_SIZE_MB
|
||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||
|
||
Manual operations:
|
||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||
```
|