807 lines
44 KiB
Markdown
807 lines
44 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
# 🖥️ UNRAID ESSENTIALS
|
||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
**System-level scripts that act on the unRAID server itself — not containers, not
|
||
media, not monitoring.** Keeping the server stable under load, recovering a frozen
|
||
WebGUI, tuning kernel limits, suppressing log noise, and handling graceful shutdowns
|
||
and reboots with proper warning sequences.
|
||
|
||
```
|
||
unRAID_Essentials/ ← acts on the server itself (this folder)
|
||
Docker_Essentials/ ← acts on containers
|
||
Media/ ← acts on the library
|
||
Monitors/ ← observes and reports
|
||
```
|
||
|
||
> **The escalation chain matters here.** Docker_Essentials handles container-level
|
||
> problems. unRAID_Essentials handles server-level problems. The watchdogs are
|
||
> designed to work together — docker_watchdog.sh heals containers first,
|
||
> system_watchdog.sh reboots only when healing has failed.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
---
|
||
|
||
### 🔴 Server Getting Into Unstable States With No Recovery Path
|
||
|
||
A container has a severe memory leak. RAM drops to 2GB. The system starts swapping.
|
||
Everything slows down. Docker watchdog tries to restart the container — but Docker
|
||
itself is barely responding. The restart hangs. The watchdog is stuck. Nothing is
|
||
getting better. The server needs a reboot, but nothing in the ecosystem is authorised
|
||
to call one.
|
||
|
||
Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The WebGUI
|
||
shows nothing useful. The server is functionally dead but still technically running.
|
||
Again — needs a reboot, nothing calls one.
|
||
|
||
The fix: `system_watchdog.sh` — the last line of defense. Three-tier response:
|
||
immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, and a
|
||
strike system for sustained threshold breaches. When everything else has failed,
|
||
system_watchdog reboots cleanly before a hard crash happens.
|
||
|
||
---
|
||
|
||
### 🔴 WebGUI Freezing and Nobody Noticing
|
||
|
||
The WebGUI becomes unresponsive. Nginx gets into a bad state. Or PHP-FPM workers are
|
||
saturated and new requests are queueing indefinitely. Or emhttp itself has frozen.
|
||
From a user perspective: dashboard doesn't load, settings don't save, containers
|
||
can't be started or stopped via the UI.
|
||
|
||
Nothing in the container stack alerts on a frozen WebGUI — it's not a container
|
||
problem, it's a web server problem. The only way to know is if someone tries to use
|
||
the UI and notices. By which point it may have been broken for hours.
|
||
|
||
The fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
|
||
php-fpm → emhttp. Lightest fix first. Notifies on any restart so you know it happened.
|
||
Silent when healthy — 144 runs per day with no output is the correct behaviour.
|
||
|
||
---
|
||
|
||
### 🔴 50+ Containers Starting and Filling Syslog With Veth Noise
|
||
|
||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual network
|
||
interface for each one. Each creation generates multiple syslog entries. In the first
|
||
few minutes after array start the syslog is buried under hundreds of lines of:
|
||
|
||
```
|
||
kernel: veth2a3b4c5: renamed from eth0
|
||
kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||
kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||
```
|
||
|
||
Real events — a failed mount, a permission error, a service that didn't start —
|
||
are invisible in this noise. And on a busy server that restarts containers regularly,
|
||
this noise continues throughout the day.
|
||
|
||
The fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any
|
||
container starts. Applied at array start, idempotent, silent when already correct.
|
||
|
||
---
|
||
|
||
### 🔴 WebGUI Queueing Requests Under Load Without Explanation
|
||
|
||
The WebGUI feels slow. Clicking a button takes 5 seconds. Saving settings seems to
|
||
hang. Nothing in the logs explains it. Container starts from the UI timeout. The
|
||
server itself is not under load — CPU is fine, RAM is fine.
|
||
|
||
The cause: PHP-FPM's `pm.max_children` defaults to 4-8 workers. On a server with
|
||
multiple users, active plugins, automated tools polling the API, and 50+ containers
|
||
all potentially hitting the WebGUI simultaneously, those 4-8 workers saturate
|
||
immediately. New requests queue behind active ones. Everything feels slow.
|
||
|
||
The fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||
250 workers × ~40MB = ~10GB worst case. On 128GB this is trivially small. The
|
||
WebGUI becomes responsive immediately. Idempotent — silent when already correct.
|
||
|
||
---
|
||
|
||
### 🔴 inotify Exhaustion Producing Unexplained Failures
|
||
|
||
Already documented in README-Monitors.md (system_tuning_monitor.sh section). Short
|
||
version: when inotify limits are exhausted, containers silently stop receiving file
|
||
system events. Downloads complete but arrs don't detect them. The kernel hits the
|
||
limit and new watches fail silently. VSCode shows "unable to watch for file changes"
|
||
and misses edits.
|
||
|
||
The fix: `inotify_tuning.sh` — raises all three inotify limits at array start.
|
||
1M watches (raised from 512K — Code-Server with node_modules needs this), 1024
|
||
instances, 32768 queued events. The startup race note: if Code-Server starts before
|
||
this runs, it inherits old limits. Restart Code-Server if the VSCode error appears
|
||
after limits are applied.
|
||
|
||
---
|
||
|
||
### 🔴 Mover Getting Killed Mid-Transfer Leaving Files Inconsistent
|
||
|
||
The mover is running — moving a large batch of files from cache to array. Someone
|
||
clicks reboot from the UI. Or a script kills the mover process directly. The mover
|
||
stops mid-file. The file exists partially on both cache and array simultaneously.
|
||
unRAID's deduplication layer is confused. The file is inaccessible.
|
||
|
||
The fix: `mover_stop.sh` — warns logged-in users via wall message, waits the
|
||
configured timeout, then sends SIGTERM (graceful) and verifies. The mover gets to
|
||
finish its current file operation before stopping. SIGKILL is a last resort with a
|
||
warning that partial files may exist.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Four distinct server-level roles:
|
||
|
||
```
|
||
🛡️ Last-resort stability system_watchdog.sh — reboots before crash
|
||
🌐 WebGUI availability webgui_restart.sh — recovers frozen UI
|
||
⚙️ Kernel tuning inotify_tuning.sh — file watch limits
|
||
php_fpm_max_children.sh — PHP worker count
|
||
🔇 Log hygiene docker_syslog_filter.sh — suppress veth noise
|
||
clear_logs.sh — weekly log trimming
|
||
🔄 Graceful operations mover_stop.sh — clean mover stop
|
||
server_reboot.sh — clean reboot with warning
|
||
user_scripts_stop.sh — stop running scripts
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
| Script | Purpose | When |
|
||
|--------|---------|------|
|
||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop |
|
||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | Every 10 minutes |
|
||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start |
|
||
| `php_fpm_max_children.sh` | Set PHP-FPM worker count | At array start |
|
||
| `docker_syslog_filter.sh` | Suppress Docker veth log noise | At array start |
|
||
| `clear_logs.sh` | Size-threshold weekly log cleanup | Weekly via maintenance window |
|
||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual |
|
||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual or called by system_watchdog |
|
||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual or called by server_reboot |
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🛡️ system_watchdog.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
The last line of defense. Reboots the system cleanly before it crashes uncleanly.
|
||
Three-tier response system — critical failures bypass everything and reboot immediately,
|
||
OOM-confirmed crises bypass the strike system, sustained threshold breaches use strikes.
|
||
Runs continuously as a background process started by `array_start.sh`.
|
||
|
||
> Full architecture documentation in `README-Docker_Essentials.md` under
|
||
> "Relationship to System Watchdog". This section covers the system_watchdog itself.
|
||
|
||
```bash
|
||
# Started by: array_start.sh (continuous background process)
|
||
# Interval: SYSTEM_WATCHDOG_INTERVAL=300 (5 minutes)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Three-Tier Response System ──────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# TIER 1 — CRITICAL (bypass ALL strikes, reboot immediately)
|
||
# These failures are acute — the system is not recoverable by waiting.
|
||
# Single detection = immediate reboot. No confirmation window.
|
||
#
|
||
# Docker daemon unresponsive:
|
||
# Attempt /etc/rc.d/rc.docker restart first.
|
||
# Wait 15 seconds. Verify daemon responding.
|
||
# If still hung → CRITICAL reboot.
|
||
# A hung daemon cannot be healed — every subsequent docker command hangs.
|
||
#
|
||
# rootfs at ROOTFS_CRITICAL_PCT (99%+):
|
||
# Writes are failing. SSH may stop. Logs can't be written.
|
||
# Nothing can be fixed from this state without a reboot.
|
||
#
|
||
# Kernel BUG/Oops in dmesg:
|
||
# Kernel running with corrupted state.
|
||
# Delta-based: new oops since last cycle → reboot.
|
||
#
|
||
# File descriptor exhaustion (FD_CRITICAL_PCT=95%):
|
||
# New connections failing. Docker can't spawn processes. SSH drops.
|
||
#
|
||
# /boot read-only:
|
||
# State files and config writes silently failing.
|
||
# Write test on /boot every cycle.
|
||
|
||
# TIER 2 — URGENT (bypass strikes when OOM confirms crisis)
|
||
# RAM < MEM_GB (4GB) AND OOM kills >= OOM_LIMIT (3) in this cycle
|
||
# Without OOM confirmation → standard strike system applies.
|
||
# Rationale: 1-2 OOM kills = docker_watchdog.sh handles it.
|
||
# 3+ kills while RAM critical = system dying faster than watchdogs heal.
|
||
# OOM victims from dmesg included in reboot message (diagnostic context).
|
||
|
||
# TIER 3 — STANDARD (strike system — N consecutive failures → reboot)
|
||
# Everything else: RAM tiers, load, CPU temp, zombies, /var/log,
|
||
# /tmp, containers, NIC, mdstat, sshd
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── RAM Tiers ────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Three-level graduated response — not a single threshold.
|
||
#
|
||
SYS_WATCHDOG_MEM_WARN_GB=10 # warn + notify once — informational only
|
||
SYS_WATCHDOG_MEM_SHUTDOWN_GB=6 # stop non-essential containers, wait for recovery
|
||
SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (or bypass if OOM confirms)
|
||
SYS_WATCHDOG_MEM_RECOVER_GB=30 # RAM must reach this before restarting containers
|
||
#
|
||
# Containers excluded from RAM emergency shutdown — these stay running:
|
||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||
"NginxProxyManager" # external access — stop this and users lose everything
|
||
"Authelia" # auth — without this nothing is accessible
|
||
"Mariadb" # Authelia dependency
|
||
"Redis" # Authelia dependency
|
||
"Emby" # media server — Live TV buffering
|
||
"Dispatcharr" # Live TV scheduler — loses state if stopped
|
||
)
|
||
#
|
||
# Coordination with docker_watchdog.sh:
|
||
# system_watchdog writes mem_shutdown_active=true to SYS_WATCHDOG_STATE_FILE.
|
||
# docker_watchdog reads this flag and defers ALL container restart logic.
|
||
# Without this: both watchdogs fight — system_watchdog stops containers,
|
||
# docker_watchdog restarts them, RAM never recovers.
|
||
# With this: docker_watchdog stands down until mem_shutdown_active clears.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Per-Host Check Toggles ───────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master_host1.conf (all checks in master_host*.conf — not master.conf)
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Each check is independently toggleable per server.
|
||
# HOST1 and HOST2 may have different hardware and different workloads.
|
||
# detect_hosts() aliases HOST*_SYS_WATCHDOG_CHECK_* → SYS_WATCHDOG_CHECK_*
|
||
#
|
||
# Tier 1 — Critical (bypass strikes):
|
||
HOST1_SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
||
HOST1_SYS_WATCHDOG_CHECK_ROOTFS=true
|
||
HOST1_SYS_WATCHDOG_CHECK_KERNEL_OOPS=true
|
||
HOST1_SYS_WATCHDOG_CHECK_FD=true
|
||
HOST1_SYS_WATCHDOG_CHECK_BOOT=true
|
||
|
||
# Tier 2 — Urgent (OOM bypass):
|
||
HOST1_SYS_WATCHDOG_CHECK_OOM=true
|
||
HOST1_SYS_WATCHDOG_CHECK_RAM=true
|
||
|
||
# Tier 3 — Standard (strike system):
|
||
HOST1_SYS_WATCHDOG_CHECK_LOG=true
|
||
HOST1_SYS_WATCHDOG_CHECK_ARC=true # HOST1 runs ZFS — enable
|
||
HOST1_SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
||
HOST1_SYS_WATCHDOG_CHECK_LOAD=false # disabled — transcoding causes normal spikes
|
||
HOST1_SYS_WATCHDOG_CHECK_ZOMBIES=true
|
||
HOST1_SYS_WATCHDOG_CHECK_CONTAINERS=true # monitors docker_watchdog persistent skip list
|
||
HOST1_SYS_WATCHDOG_CHECK_TMP=true
|
||
HOST1_SYS_WATCHDOG_CHECK_MDSTAT=true
|
||
HOST1_SYS_WATCHDOG_CHECK_NETWORK=true
|
||
HOST1_SYS_WATCHDOG_CHECK_SSHD=true
|
||
HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false # disabled — may false positive during encoding
|
||
HOST1_SYS_WATCHDOG_NIC="eth0" # verify: ip link show | grep "^[0-9]"
|
||
```
|
||
|
||
---
|
||
|
||
### ── Abort Conditions ─────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Conditions that prevent a reboot even when a threshold is hit.
|
||
# CRITICAL tier bypasses these — truly critical conditions reboot regardless.
|
||
#
|
||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # rebooting with bad pool risks data loss
|
||
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity is better than crashing
|
||
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting mover is better than crashing
|
||
#
|
||
# Philosophy: a graceful reboot before a crash is always better than a hard crash.
|
||
# The abort conditions protect against the cases where a reboot itself causes harm
|
||
# (data loss from bad ZFS pool). Parity and mover can be restarted after reboot.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Reboot Loop Protection ───────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# If the server keeps rebooting, something is wrong that rebooting isn't fixing.
|
||
# After REBOOT_LIMIT reboots in REBOOT_WINDOW_HRS → shutdown instead.
|
||
# Shutdown prevents: hardware damage, filesystem corruption from repeated reboots,
|
||
# infinite loop that never lets you investigate.
|
||
# State file: /boot/config/system_watchdog_reboots.db — survives reboots.
|
||
#
|
||
SYS_WATCHDOG_REBOOT_LIMIT=3 # reboots before shutdown instead
|
||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12 # rolling window in hours
|
||
```
|
||
|
||
---
|
||
|
||
### ── State File Heartbeat ─────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# system_watchdog writes watchdog_cycle=N to SYS_WATCHDOG_STATE_FILE every cycle.
|
||
# This keeps the file's modification time current.
|
||
#
|
||
# docker_watchdog.sh uses the state file mtime as a stale guard — if the file
|
||
# is more than 2 hours old while mem_shutdown_active=true is set, system_watchdog
|
||
# may have stopped running. docker_watchdog resumes normal operation rather than
|
||
# being silenced indefinitely by a stale flag.
|
||
#
|
||
# Without this heartbeat: if all standard checks pass and no state writes happen
|
||
# (e.g. CHECK_KERNEL_OOPS=false AND CHECK_MDSTAT=false), the file mtime could go
|
||
# stale even with the watchdog running.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
system_watchdog.sh # normal (continuous — started by array_start.sh)
|
||
system_watchdog.sh --dry-run # trigger detection without rebooting
|
||
system_watchdog.sh --status # show all tiers, thresholds, active strikes, RAM state
|
||
system_watchdog.sh --log # verbose per-cycle output
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🌐 webgui_restart.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Monitors the unRAID WebGUI availability and recovers it automatically when unresponsive.
|
||
Three-step escalating strategy — lightest fix first, heaviest last. Silent when healthy.
|
||
|
||
```bash
|
||
# Scheduled: */10 * * * * (every 10 minutes)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Three-Step Escalation ────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Step 1 — nginx restart
|
||
# Lightest fix — handles most WebGUI failures.
|
||
# nginx is the web server layer. Crash, worker stuck, connection timeout.
|
||
# Wait WEBGUI_NGINX_WAIT seconds → curl recheck.
|
||
#
|
||
# Step 2 — php-fpm restart
|
||
# Added because WebGUI can appear frozen due to PHP worker exhaustion.
|
||
# pm.max_children workers all occupied → new requests queue → dashboard hangs.
|
||
# php-fpm restart far less disruptive than emhttp.
|
||
# Wait WEBGUI_PHP_WAIT seconds → curl recheck.
|
||
# Note: system_tuning_monitor.sh tracks worker saturation over time.
|
||
#
|
||
# Step 3 — emhttp restart
|
||
# Heaviest fix. emhttp is the core unRAID management daemon.
|
||
# Array, Docker, shares continue running — only WebGUI management restarts.
|
||
# Takes longer to recover — WEBGUI_EMHTTP_WAIT gives it time.
|
||
# Wait WEBGUI_EMHTTP_WAIT seconds → curl recheck.
|
||
#
|
||
# If all three fail → notify warning — manual intervention needed.
|
||
# Guidance: pgrep -x nginx emhttp | check journalctl | consider server_reboot.sh
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Configuration ────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
WEBGUI_URL="http://localhost" # adjust if non-standard port
|
||
WEBGUI_TIMEOUT=5 # seconds before curl times out
|
||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart — takes longer
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
webgui_restart.sh # check once — silent if healthy, escalates if not
|
||
webgui_restart.sh --dry-run # walk through escalation without restarting anything
|
||
webgui_restart.sh --status # show WebGUI state + nginx/php-fpm/emhttp process state
|
||
webgui_restart.sh --log # verbose — show each escalation step
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 📡 inotify_tuning.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Raises Linux inotify kernel limits at array start. Idempotent — completely silent
|
||
when values are already correct.
|
||
|
||
```bash
|
||
# Scheduled: At Startup of Array (via array_start.sh — FIRST in ARRAY_START_SCRIPTS)
|
||
# Must run before containers start — containers inherit limits at startup
|
||
```
|
||
|
||
---
|
||
|
||
### ── Three Limits ─────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
INOTIFY_MAX_INSTANCES=1024 # default: 128 — max inotify instances per user
|
||
# 1024 handles ~20-30 containers watching files
|
||
|
||
INOTIFY_MAX_WATCHES=1048576 # default: 8192 — SHARED budget across ALL users/containers
|
||
# Was 512K — raised to 1M (1048576)
|
||
# VSCode/Code-Server alone needs ~50K-200K for large workspaces
|
||
# with node_modules. All arr containers + Emby + VSCode share
|
||
# this budget. 1M safe on 128GB RAM (~128MB kernel memory)
|
||
# If VSCode shows "unable to watch for file changes" → too low
|
||
|
||
INOTIFY_MAX_QUEUED_EVENTS=32768 # default: 16384 — events buffered before dropping
|
||
```
|
||
|
||
---
|
||
|
||
### ── Startup Race Condition ───────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# inotify limits are kernel-wide — they take effect immediately on sysctl write.
|
||
# But containers that have ALREADY started inherit the OLD limits at startup.
|
||
# Those containers keep their inherited (low) limits until restarted.
|
||
#
|
||
# This is why inotify_tuning.sh must be FIRST in ARRAY_START_SCRIPTS — before
|
||
# any container starts. If Code-Server starts before limits are raised:
|
||
# → Code-Server inherits old 8192 watch limit
|
||
# → VSCode shows "unable to watch for file changes"
|
||
# → Fix: docker restart Code-Server (picks up current kernel limits on start)
|
||
#
|
||
# The script warns if it changed any values:
|
||
# "If Code-Server is running: docker restart Code-Server"
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
inotify_tuning.sh # normal run (idempotent — silent when correct)
|
||
inotify_tuning.sh --dry-run # show what would change without changing
|
||
inotify_tuning.sh --status # current values vs targets + top inotify consumers
|
||
inotify_tuning.sh --log # verbose — show each sysctl write
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## ⚙️ php_fpm_max_children.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Raises PHP-FPM `pm.max_children` at array start to prevent WebGUI queueing under
|
||
load. Idempotent — completely silent when already correct. No PHP-FPM restart unless
|
||
value actually changed.
|
||
|
||
```bash
|
||
# Scheduled: At Startup of Array (via array_start.sh)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Why 250 Workers ──────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
PHP_MAX_CHILDREN=250 # default: 4-8 — far too low for a busy server
|
||
# Each worker: ~2MB resident memory at idle
|
||
# 250 workers × 2MB = ~500MB — trivial on 128GB
|
||
# Worst case (all active): ~250 × 40MB = ~10GB
|
||
# In practice: rarely all active simultaneously
|
||
# On 64GB (HOST2): still appropriate — 250 × 40MB
|
||
# = 10GB worst case = 15% of RAM, acceptable
|
||
#
|
||
# Without this fix:
|
||
# 5 users hit the WebGUI simultaneously → 8 workers exhausted → 5 more queue
|
||
# Each queued request waits for a worker to free → 5-10 second response times
|
||
# Looks like a slow server — it's just a queue
|
||
#
|
||
PHP_CONF="/etc/php83/php-fpm.d/www.conf" # path may change with PHP version
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
php_fpm_max_children.sh # normal run (idempotent)
|
||
php_fpm_max_children.sh --dry-run # show what would change
|
||
php_fpm_max_children.sh --status # show current value vs target + worker count
|
||
php_fpm_max_children.sh --log # verbose — show config write + restart
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔇 docker_syslog_filter.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Creates an rsyslog drop rule for Docker veth/docker0 interface noise. Idempotent —
|
||
silent when filter already correct, only writes + restarts rsyslog when something changed.
|
||
|
||
```bash
|
||
# Scheduled: At Startup of Array (via array_start.sh — before containers start)
|
||
```
|
||
|
||
---
|
||
|
||
### ── What Gets Suppressed ─────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# These kernel messages are generated on every container start and stop:
|
||
#
|
||
# kernel: veth2a3b4c5: renamed from eth0
|
||
# kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||
# kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||
# kernel: docker0: port 1(veth2a3b4c5) entered disabled state
|
||
#
|
||
# 50+ containers at array start = 200-400 lines of this in the first minute.
|
||
# Containers restart throughout the day = continuous noise.
|
||
# Real events buried and invisible in syslog.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Idempotent Design ────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Runs at every array start but only changes something when needed:
|
||
# Filter file exists and content is correct → exit 0 silently
|
||
# Filter file missing or content changed → write + restart rsyslog
|
||
#
|
||
# Expected content compared exactly — single source of truth:
|
||
EXPECTED_FILTER='if ($msg contains "veth" or $msg contains "docker0") then {
|
||
stop
|
||
}'
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
docker_syslog_filter.sh # normal run (idempotent — silent when correct)
|
||
docker_syslog_filter.sh --dry-run # show what would be written without writing
|
||
docker_syslog_filter.sh --status # show current filter file + rsyslog state
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🗑️ clear_logs.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Clears system logs and Docker container logs using size thresholds. Only clears logs
|
||
large enough to be worth clearing — preserves recent diagnostic context on small logs.
|
||
|
||
```bash
|
||
# Called by: weekly_sync_maintenance.sh via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Size Threshold Approach ──────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Why thresholds instead of clearing everything:
|
||
# A 2MB syslog contains useful recent history — not worth clearing.
|
||
# A 500MB Docker log is consuming rootfs and contains mostly noise — clear it.
|
||
# Blind truncation destroys diagnostic context for no benefit.
|
||
#
|
||
LOG_MIN_SIZE_MB=10 # skip system log if under this size — keep history
|
||
LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this size
|
||
# Active containers (Emby, SABnzbd) grow fastest
|
||
# 100MB × 30 containers = 3GB before any clearing kicks in
|
||
|
||
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
|
||
#
|
||
# Truncation not logrotate:
|
||
# unRAID writes to tmpfs (/var/log) — logrotate's compress + archive approach
|
||
# would consume more tmpfs space, not less.
|
||
# : > file keeps the file descriptor valid while emptying content.
|
||
# Safe for running services (syslogd continues writing to the same fd).
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
clear_logs.sh # normal run — silent if all logs under threshold
|
||
clear_logs.sh --dry-run # show what would be cleared and sizes
|
||
clear_logs.sh --status # show current log sizes vs thresholds
|
||
clear_logs.sh --log # verbose — show each file evaluated
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## ⏹️ mover_stop.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Stops the unRAID mover cleanly — wall warning, configurable timeout, SIGTERM → verify
|
||
→ SIGKILL sequence. Safe to run when mover is not running — exits cleanly with a log.
|
||
|
||
---
|
||
|
||
### ── Stop Sequence ────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# 1. Check if mover is running — exit cleanly if not
|
||
# 2. Wall message: "HOST1 (unRAID-Gmer4Lfe) — mover stopping in 30s"
|
||
# MY_ID included — on shared terminal it's clear which server
|
||
# 3. Wait MOVER_STOP_TIMEOUT seconds
|
||
# 4. SIGTERM — allows mover to finish its current file before stopping
|
||
# No partial files — the mover completes what it's working on
|
||
# 5. Wait 5 seconds — verify if stopped
|
||
# 6. If still running → SIGKILL (forced)
|
||
# Warning: partial files possible — same as a hard crash
|
||
# 7. Final verify — error if still running
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
mover_stop.sh # stop mover with configured timeout
|
||
mover_stop.sh --dry-run # show what would happen
|
||
mover_stop.sh --status # show mover state (running, PID, start time)
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔁 server_reboot.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Graceful reboot with pre-flight warnings, wall message, unRAID notification, VM
|
||
graceful shutdown, then Docker and services stop, sync, reboot.
|
||
|
||
> Full documentation in `README-Tools.md` — `server_reboot.sh` section. This is a
|
||
> quick reference.
|
||
|
||
---
|
||
|
||
### ── Shutdown Sequence ────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# 1. Pre-flight warnings (warn not block):
|
||
# rsync running, mover running, active Emby sessions
|
||
# Warnings show in summary — you chose to reboot, these are for context
|
||
# 2. Wall message + unRAID notification — MY_ID included
|
||
# 3. Wait REBOOT_SLEEP seconds (default 30)
|
||
# 4. virsh shutdown each VM → wait REBOOT_VM_WAIT seconds for graceful exit
|
||
# 5. Stop libvirt (VM Manager)
|
||
# 6. Stop Docker service
|
||
# 7. sync — flush filesystem buffers
|
||
# 8. /sbin/reboot
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
server_reboot.sh # reboot with 30s warning
|
||
server_reboot.sh --dry-run # full sequence walkthrough without rebooting
|
||
server_reboot.sh --status # show running processes that would be affected
|
||
server_reboot.sh --reason="maintenance" # include reason in wall + notification
|
||
server_reboot.sh --log # verbose per-step output
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🛑 user_scripts_stop.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Stops all running User Script processes. Identifies by `/tmp/user.scripts` path
|
||
signature. Shows script names not PIDs. SIGTERM → verify → SIGKILL with self-exclusion.
|
||
|
||
> Full documentation in `README-Tools.md` — `user_scripts_stop.sh` section.
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
user_scripts_stop.sh # stop all — SIGTERM → SIGKILL if needed
|
||
user_scripts_stop.sh --dry-run # show which scripts would be stopped, by name
|
||
user_scripts_stop.sh --status # show running scripts with PID and runtime
|
||
user_scripts_stop.sh --log # verbose per-process output
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ STARTUP SEQUENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# master.conf — ARRAY_START_SCRIPTS (order matters)
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
ARRAY_START_SCRIPTS=(
|
||
# ── One-shot — run and exit ───────────────────────────────────────────────
|
||
"unRAID_Essentials/inotify_tuning.sh" # FIRST — raise limits before
|
||
# containers inherit old values
|
||
"unRAID_Essentials/docker_syslog_filter.sh" # SECOND — before containers
|
||
# create veth interfaces
|
||
"unRAID_Essentials/php_fpm_max_children.sh" # before WebGUI serves requests
|
||
"Transcodes/ramdisk_setup.sh" # before Emby starts transcoding
|
||
"Docker_Essentials/docker_network_connect.sh" # before watchdogs check states
|
||
|
||
# ── Continuous — run until array stops ───────────────────────────────────
|
||
"unRAID_Essentials/system_watchdog.sh" # before docker_watchdog —
|
||
# writes state file docker_watchdog reads
|
||
"Docker_Essentials/docker_watchdog.sh" # before failover — containers
|
||
# must be healthy for failover decisions
|
||
"Failover/failover.sh" # last — needs everything stable
|
||
)
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# array_start.sh is the ONLY "At Startup of Array" entry in User Scripts.
|
||
# It launches everything above in order.
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ FULL SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# At Startup of Array — via array_start.sh:
|
||
# inotify_tuning.sh
|
||
# docker_syslog_filter.sh
|
||
# php_fpm_max_children.sh
|
||
# (ramdisk_setup.sh — in Transcodes/)
|
||
# system_watchdog.sh (continuous)
|
||
|
||
# Every 10 minutes:
|
||
*/10 * * * * webgui_restart.sh # silent when healthy — escalates when not
|
||
|
||
# Weekly — via weekly_sync_maintenance.sh:
|
||
# clear_logs.sh # Sunday 2:30am via WEEKLY_MAINTENANCE_SCRIPTS
|
||
|
||
# Manual:
|
||
# mover_stop.sh — before array ops that need mover stopped
|
||
# server_reboot.sh — planned maintenance reboots
|
||
# user_scripts_stop.sh — emergency script stop or pre-reboot cleanup
|
||
``` |