Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename) - Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API for storage health, service management, mover, user scripts, notifications, disk temps, and platform command validation - Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR, auto-source Plugin/$PLATFORM/adapter.sh after common.sh - Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg, disks.ini, and validate_unraid_cmd calls with platform_*() functions across watchdogs, orchestrators, and System_Essentials scripts - Update all documentation: rename refs, update webgui escalation logic, add platform adapter section to Plugin README, update main README with portability vision and corrected self-healing stack description
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,160 @@
|
||||
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
|
||||
|
||||
**System-level scripts that act on the server itself — not containers,
|
||||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||||
shutdowns with proper warning sequences.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Server Getting Into Unstable States With No Recovery Path**
|
||||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||||
write log files. The server is functionally dead but still technically running.
|
||||
|
||||
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||||
threshold breaches. Last line of defense before a hard crash.
|
||||
|
||||
**WebGUI Freezing and Nobody Noticing**
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||||
container-level alert fires because this isn't a container problem — it's a web
|
||||
server problem. By the time someone notices it may have been broken for hours.
|
||||
|
||||
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||||
|
||||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||||
network interface for each one. Each interface generates multiple syslog entries.
|
||||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||||
|
||||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||||
starts. Applied at array start. Idempotent — silent when already correct.
|
||||
|
||||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||||
those workers saturate immediately. New requests queue behind active ones.
|
||||
|
||||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||||
|
||||
**inotify Exhaustion Producing Unexplained Failures**
|
||||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||||
and all containers share the same pool.
|
||||
|
||||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||||
|
||||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||||
array simultaneously. unRAID's deduplication layer is confused.
|
||||
|
||||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
```
|
||||
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
|
||||
Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||||
clear_logs.sh — weekly log trimming
|
||||
Graceful operations mover_stop.sh — clean mover stop
|
||||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||||
user_scripts_stop.sh — stop running User Scripts
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||||
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
Orchestrators/
|
||||
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||||
─────────────────────────► docker_syslog_filter.sh (second)
|
||||
─────────────────────────► php_fpm_max_children.sh
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
|
||||
Watchdogs/
|
||||
stability_watchdog.sh and resource_watchdog.sh now live here.
|
||||
See Watchdogs/README-Watchdogs.md for how they relate to each other
|
||||
and to docker_watchdog.sh and System/storage_watchdog.sh.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||||
└─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
|
||||
Every minute (watchdog_orchestrator.sh in Orchestrators/):
|
||||
→ Watchdogs/resource_watchdog.sh
|
||||
→ Watchdogs/docker_watchdog.sh
|
||||
→ Watchdogs/system_watchdog.sh (thin orchestrator)
|
||||
└─ Watchdogs/System/storage_watchdog.sh
|
||||
└─ Watchdogs/System/webgui_watchdog.sh
|
||||
WebGUI OK → silent exit
|
||||
Not responding:
|
||||
Step 1: restart nginx → recheck
|
||||
Step 2: restart php-fpm → recheck
|
||||
Step 3: restart emhttp → recheck
|
||||
All failed → notify, exit 1
|
||||
└─ Watchdogs/System/network_watchdog.sh
|
||||
→ Watchdogs/stability_watchdog.sh
|
||||
(see Watchdogs/README-Watchdogs.md for full flow)
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
System logs: clear if > LOG_MIN_SIZE_MB
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
Manual operations:
|
||||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||||
```
|
||||
Reference in New Issue
Block a user