Gmer4Lfe
e13f2fa14f
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
177 lines
8.9 KiB
Markdown
177 lines
8.9 KiB
Markdown
# ━━━━━ UNRAID ESSENTIALS ━━━━━
|
||
|
||
**System-level scripts that act on the unRAID server itself — not containers,
|
||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||
shutdowns with proper warning sequences.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||
|
||
**Server Getting Into Unstable States With No Recovery Path**
|
||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||
write log files. The server is functionally dead but still technically running.
|
||
|
||
Fix: `system_watchdog.sh` — three-tier response: immediate reboot on critical
|
||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||
threshold breaches. Last line of defense before a hard crash.
|
||
|
||
**WebGUI Freezing and Nobody Noticing**
|
||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||
container-level alert fires because this isn't a container problem — it's a web
|
||
server problem. By the time someone notices it may have been broken for hours.
|
||
|
||
Fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
|
||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||
|
||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||
network interface for each one. Each interface generates multiple syslog entries.
|
||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||
|
||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||
starts. Applied at array start. Idempotent — silent when already correct.
|
||
|
||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||
those workers saturate immediately. New requests queue behind active ones.
|
||
|
||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||
|
||
**inotify Exhaustion Producing Unexplained Failures**
|
||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||
and all containers share the same pool.
|
||
|
||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||
|
||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||
array simultaneously. unRAID's deduplication layer is confused.
|
||
|
||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||
|
||
```
|
||
Last-resort stability system_watchdog.sh — reboots before crash
|
||
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
|
||
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
|
||
Kernel tuning inotify_tuning.sh — file watch limits
|
||
php_fpm_max_children.sh — PHP worker count
|
||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||
clear_logs.sh — weekly log trimming
|
||
Graceful operations mover_stop.sh — clean mover stop
|
||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||
user_scripts_stop.sh — stop running User Scripts
|
||
server_reboot.sh — clean reboot with pre-flight warnings
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||
|
||
```
|
||
Orchestrators/
|
||
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||
─────────────────────────► docker_syslog_filter.sh (second)
|
||
─────────────────────────► php_fpm_max_children.sh
|
||
─────────────────────────► system_watchdog.sh (background loop)
|
||
|
||
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
|
||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||
|
||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||
|
||
Docker_Essentials/
|
||
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
|
||
(mem_shutdown_active flag)
|
||
```
|
||
|
||
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
|
||
designed to work together — docker_watchdog heals containers first,
|
||
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
|
||
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
|
||
prevent docker_watchdog from restarting containers that resource_watchdog just
|
||
stopped to free RAM.
|
||
|
||
---
|
||
|
||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||
|
||
| Script | Role | When It Runs |
|
||
|--------|------|-------------|
|
||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
|
||
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
|
||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
|
||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||
|
||
---
|
||
|
||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||
|
||
```
|
||
Array starts
|
||
│
|
||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||
├─ php_fpm_max_children.sh ← before WebGUI is under load
|
||
└─ system_watchdog.sh ← starts background loop
|
||
|
||
|
||
Every minute (watchdog_orchestrator.sh):
|
||
└─ resource_watchdog.sh
|
||
Level 1 (soft): throttle SABnzbd + qBit download speeds
|
||
Level 2 (medium): further throttle + docker pause non-essential containers
|
||
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
|
||
↓
|
||
docker_watchdog.sh reads mem_shutdown_active — defers restarts
|
||
|
||
|
||
Every 10 minutes (User Scripts):
|
||
└─ webgui_restart.sh
|
||
WebGUI OK → silent exit
|
||
Not responding:
|
||
Step 1: restart nginx → recheck
|
||
Step 2: restart php-fpm → recheck
|
||
Step 3: restart emhttp → recheck
|
||
All failed → notify, exit 1
|
||
|
||
|
||
Weekly (weekly_maintenance.sh):
|
||
└─ clear_logs.sh
|
||
System logs: clear if > LOG_MIN_SIZE_MB
|
||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||
|
||
|
||
Manual operations:
|
||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||
```
|