feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,768 @@
|
||||
# ━━━━━ UNRAID ESSENTIALS — Manual ━━━━━
|
||||
|
||||
Configuration reference, operational procedures, and troubleshooting for
|
||||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||||
adding or reordering scripts at array start.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||||
- [webgui_restart.sh](#webgui_restartsh)
|
||||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||||
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
|
||||
- [clear_logs.sh](#clear_logssh)
|
||||
- [mover_stop.sh](#mover_stopsh)
|
||||
- [rsync_stop.sh](#rsync_stopsh)
|
||||
- [user_scripts_stop.sh](#user_scripts_stopsh)
|
||||
- [server_reboot.sh](#server_rebootsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## ARRAY_START_SCRIPTS Order
|
||||
|
||||
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
|
||||
> scripts.** Getting it wrong causes subtle failures that don't show up
|
||||
> immediately.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
|
||||
# any container starts. Containers inherit
|
||||
# inotify limits at launch, not dynamically.
|
||||
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
|
||||
# created. If a container starts first, its
|
||||
# veth creation is already in syslog.
|
||||
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
|
||||
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
|
||||
...
|
||||
"system_watchdog.sh" # LAST or near-last — starts background loop
|
||||
)
|
||||
```
|
||||
|
||||
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
|
||||
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
|
||||
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
|
||||
|
||||
Why docker_syslog_filter SECOND: The filter must be in place before any container
|
||||
starts creating veth interfaces. The first container start after array start
|
||||
generates veth messages — these will appear in syslog if the filter isn't active.
|
||||
|
||||
---
|
||||
|
||||
## system_watchdog.sh
|
||||
|
||||
### Three-Tier Response
|
||||
|
||||
The watchdog categorizes every failure into one of three tiers:
|
||||
|
||||
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
|
||||
| Condition | Threshold | Why immediate |
|
||||
|-----------|-----------|---------------|
|
||||
| Docker daemon unresponsive | N/A | Nothing can be healed; every docker command hangs |
|
||||
| rootfs usage | SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) | SSH stops; state files fail silently |
|
||||
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
|
||||
| File descriptor exhaustion | SYS_WATCHDOG_FD_CRITICAL_PCT (95%) | New connections silently failing |
|
||||
| /boot read-only unexpectedly | write test fails | Config writes silently failing |
|
||||
|
||||
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
|
||||
|
||||
RAM below MEM_GB AND OOM kills this cycle >= SYS_WATCHDOG_OOM_LIMIT.
|
||||
Both conditions required — RAM alone without OOM uses the standard strike system.
|
||||
OOM confirms the system is dying faster than watchdogs can heal.
|
||||
|
||||
**Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)**
|
||||
| Check | Threshold |
|
||||
|-------|-----------|
|
||||
| Free RAM | MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB |
|
||||
| Load average | SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count |
|
||||
| CPU temperature | SYS_WATCHDOG_CPU_TEMP |
|
||||
| Zombie processes | SYS_WATCHDOG_ZOMBIES |
|
||||
| /var/log usage | SYS_WATCHDOG_VAR_LOG_PCT |
|
||||
| /tmp usage | SYS_WATCHDOG_TMP_PCT |
|
||||
| Array disk errors | mdstat error delta > 0 |
|
||||
| NIC state | interface operstate != "up" |
|
||||
| Required containers | containers in SYS_WATCHDOG_REQUIRED_CONTAINERS |
|
||||
|
||||
### RAM Tiers
|
||||
|
||||
```
|
||||
MEM_WARN_GB (10GB) → warn + notify, no action
|
||||
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
|
||||
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
|
||||
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
|
||||
```
|
||||
|
||||
At MEM_SHUTDOWN_GB, all containers NOT listed in
|
||||
`SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` are stopped. Excluded containers by default:
|
||||
NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr. Adjust in
|
||||
master.conf for your critical services.
|
||||
|
||||
### Abort Conditions
|
||||
|
||||
These conditions prevent a reboot — running them would cause data loss:
|
||||
|
||||
```bash
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
|
||||
```
|
||||
|
||||
CRITICAL tier bypasses all abort conditions — an imminent crash outweighs
|
||||
data safety concerns.
|
||||
|
||||
### Strike System
|
||||
|
||||
The strike count tracks consecutive failures. A single recovery resets strikes
|
||||
to 0. The reboot only fires after SYS_WATCHDOG_STRIKES consecutive failures on
|
||||
the same check — transient spikes (a brief load burst, a momentary RAM dip) don't
|
||||
trigger reboots.
|
||||
|
||||
### Reboot Rate Limit
|
||||
|
||||
```bash
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
|
||||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
|
||||
```
|
||||
|
||||
If the server has rebooted SYS_WATCHDOG_MAX_REBOOTS times within the window,
|
||||
system_watchdog stops rebooting and notifies instead. This prevents a boot loop
|
||||
where the watchdog reboots → something crashes again immediately → reboot again.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # start continuous monitoring loop
|
||||
system_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
system_watchdog.sh --status # show thresholds, current state, strike counts
|
||||
system_watchdog.sh --log # verbose per-cycle output
|
||||
```
|
||||
|
||||
### Verify Running
|
||||
|
||||
```bash
|
||||
pgrep -a -f system_watchdog.sh
|
||||
# Expected: shows PID and path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## resource_watchdog.sh
|
||||
|
||||
### Pressure Levels
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
|
||||
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
|
||||
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers
|
||||
RW_RAM_RECOVER_GB=25 # recover to this before un-stopping at level 3
|
||||
|
||||
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
|
||||
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
|
||||
|
||||
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
|
||||
```
|
||||
|
||||
### Per-Host Container Lists
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure
|
||||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure
|
||||
```
|
||||
|
||||
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
|
||||
pressure level. Default includes: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
|
||||
|
||||
### docker_watchdog Coordination
|
||||
|
||||
At level 3, resource_watchdog writes `mem_shutdown_active=true` to `RW_STATE_FILE`.
|
||||
docker_watchdog.sh reads this flag each cycle and skips all container restart logic
|
||||
while it is set. Without this coordination, docker_watchdog would immediately
|
||||
restart containers that resource_watchdog just stopped to free RAM.
|
||||
|
||||
The flag is cleared when level 3 pressure resolves and containers are restarted.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
|
||||
resource_watchdog.sh --status # current level, active actions, recovery cycle count
|
||||
resource_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## webgui_restart.sh
|
||||
|
||||
### Escalation Logic
|
||||
|
||||
```
|
||||
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
|
||||
|
||||
Not responding:
|
||||
1. /etc/rc.d/rc.nginx restart
|
||||
wait WEBGUI_NGINX_WAIT (15s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
2. /etc/rc.d/rc.php-fpm restart
|
||||
wait WEBGUI_PHP_WAIT (10s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
3. /usr/local/sbin/emhttp stop && start
|
||||
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
All three failed → notify warning, exit 1
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
WEBGUI_URL="http://localhost" # URL to check
|
||||
WEBGUI_TIMEOUT=5 # curl timeout in seconds
|
||||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
|
||||
```
|
||||
|
||||
### WebGUI Frozen — Manual Recovery
|
||||
|
||||
```bash
|
||||
# Check which services are running:
|
||||
webgui_restart.sh --status
|
||||
|
||||
# Try manual restart sequence (same as the script):
|
||||
/etc/rc.d/rc.nginx restart
|
||||
# wait 15s, then:
|
||||
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
|
||||
|
||||
# If nginx didn't fix it, php-fpm:
|
||||
/etc/rc.d/rc.php-fpm restart
|
||||
|
||||
# If still down, emhttp:
|
||||
/usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
|
||||
|
||||
# If all three failed:
|
||||
server_reboot.sh --status # check for active sessions first
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## inotify_tuning.sh
|
||||
|
||||
### What It Sets
|
||||
|
||||
```bash
|
||||
INOTIFY_MAX_INSTANCES=1024 # max inotify fd objects per user (default: 128)
|
||||
INOTIFY_MAX_WATCHES=1048576 # max watches shared across all users (default: 8192)
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768 # max buffered events (default: 16384)
|
||||
```
|
||||
|
||||
Verify current values:
|
||||
|
||||
```bash
|
||||
inotify_tuning.sh --status
|
||||
# Shows current vs target for each limit, active instance count, top consumers
|
||||
```
|
||||
|
||||
### If Code-Server Shows "Unable to Watch for File Changes"
|
||||
|
||||
```bash
|
||||
# 1. Verify limits are set:
|
||||
sysctl fs.inotify.max_user_watches
|
||||
# Expected: 1048576
|
||||
|
||||
# 2. Check total usage across all containers:
|
||||
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l
|
||||
|
||||
# 3. If limits are set but Code-Server still shows the error:
|
||||
docker restart Code-Server
|
||||
# Running containers inherit limits at launch. Restart picks up the new values.
|
||||
|
||||
# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
|
||||
inotify_tuning.sh --log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## php_fpm_max_children.sh
|
||||
|
||||
### What It Sets
|
||||
|
||||
```bash
|
||||
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
```
|
||||
|
||||
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
|
||||
The default of 4–8 saturates immediately under load on a busy server.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
php_fpm_max_children.sh --status
|
||||
# Shows current value vs target, PHP-FPM worker count
|
||||
|
||||
# Manual verify:
|
||||
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
|
||||
# Expected: pm.max_children = 250
|
||||
```
|
||||
|
||||
### If WebGUI Is Slow Despite the Setting
|
||||
|
||||
```bash
|
||||
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
|
||||
# Look at the webgui_restart.sh escalation — step 2 (php-fpm restart) is specifically
|
||||
# for worker exhaustion. If webgui_restart.sh is regularly hitting step 2, the
|
||||
# pm.max_children value may still be too low, or there's a PHP worker leak.
|
||||
|
||||
# Check running worker count:
|
||||
pgrep -fc php-fpm
|
||||
# Compare to pm.max_children — if equal, workers are saturated
|
||||
|
||||
# Increase if needed:
|
||||
# master.conf: PHP_MAX_CHILDREN=350
|
||||
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## docker_syslog_filter.sh
|
||||
|
||||
### What It Creates
|
||||
|
||||
```
|
||||
/etc/rsyslog.d/ignore-docker-veth.conf:
|
||||
if ($msg contains "veth" or $msg contains "docker0") then {
|
||||
stop
|
||||
}
|
||||
```
|
||||
|
||||
This drops any syslog message containing "veth" or "docker0" before it reaches
|
||||
any output target, including the log file. The drop rule is applied at rsyslog
|
||||
level — not at the log viewer level.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
docker_syslog_filter.sh --status
|
||||
# Shows filter file content and rsyslog process state
|
||||
|
||||
# Manual verify:
|
||||
cat /etc/rsyslog.d/ignore-docker-veth.conf
|
||||
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"
|
||||
|
||||
# Test the filter is active (should produce no syslog output):
|
||||
logger "test veth message"
|
||||
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
|
||||
```
|
||||
|
||||
### If Syslog Still Has Veth Noise
|
||||
|
||||
```bash
|
||||
# 1. Verify filter file exists with correct content:
|
||||
docker_syslog_filter.sh --status
|
||||
|
||||
# 2. If content differs — re-apply:
|
||||
docker_syslog_filter.sh --log
|
||||
|
||||
# 3. Verify rsyslog is using the conf.d directory:
|
||||
grep -r "IncludeConfig" /etc/rsyslog.conf
|
||||
# Expected: IncludeConfig /etc/rsyslog.d/*.conf (or similar)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## clear_logs.sh
|
||||
|
||||
### Thresholds
|
||||
|
||||
```bash
|
||||
LOG_MIN_SIZE_MB=10 # skip system log if under this — keep recent history
|
||||
LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this
|
||||
|
||||
LOG_FILES=(
|
||||
"/var/log/syslog"
|
||||
"/var/log/messages"
|
||||
"/var/log/dmesg"
|
||||
)
|
||||
```
|
||||
|
||||
### Why Truncation Not Deletion
|
||||
|
||||
unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
|
||||
descriptor open and valid while emptying content — syslogd continues writing to
|
||||
the same fd without interruption. Deleting the file would orphan the file
|
||||
descriptor and syslog would stop writing until restarted.
|
||||
|
||||
### Identifying Large Docker Logs
|
||||
|
||||
```bash
|
||||
clear_logs.sh --status
|
||||
# Shows top 10 Docker logs by size, current size vs threshold
|
||||
|
||||
# Find the biggest log manually:
|
||||
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## mover_stop.sh
|
||||
|
||||
### Stop Sequence
|
||||
|
||||
```
|
||||
1. Check if mover is running (pgrep "emhttp.*Mover") → exit cleanly if not
|
||||
2. Wall message to all logged-in terminal users
|
||||
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
|
||||
4. SIGTERM — mover finishes its current file operation, then stops
|
||||
5. Wait 5 seconds → verify stopped
|
||||
6. SIGKILL if still running — forced stop, partial files possible
|
||||
7. Final verify — error if still running after SIGKILL
|
||||
```
|
||||
|
||||
SIGTERM first because the mover can finish the file it is currently moving,
|
||||
leaving no partial copies split across cache and array. SIGKILL is a last resort.
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
mover_stop.sh # check and stop if running
|
||||
mover_stop.sh --status # show current mover state and PID
|
||||
mover_stop.sh --dry-run # show what would happen without stopping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## rsync_stop.sh
|
||||
|
||||
### Auto-Detection Logic
|
||||
|
||||
rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
|
||||
is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.
|
||||
|
||||
**Default behavior (orchestrator detected):**
|
||||
Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
|
||||
next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.
|
||||
|
||||
**Default behavior (no orchestrator):**
|
||||
Kills rsync directly (standalone rsync.sh run).
|
||||
|
||||
**--full-stop:**
|
||||
Kills the orchestrator first, then kills rsync. Nothing continues after this.
|
||||
Use when everything needs to stop immediately.
|
||||
|
||||
### Container Recovery
|
||||
|
||||
After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
|
||||
for any that were stopped by the interrupted rsync session and restarts them.
|
||||
Remote containers are left for docker_watchdog.sh to recover.
|
||||
|
||||
Skip container recovery with `--rsync-only` — used when called by other scripts
|
||||
that handle recovery themselves.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # smart stop (auto-detect orchestrator)
|
||||
rsync_stop.sh --full-stop # kill orchestrator + rsync
|
||||
rsync_stop.sh --rsync-only # kill rsync, skip container recovery
|
||||
rsync_stop.sh --status # show local and remote rsync state
|
||||
rsync_stop.sh --dry-run # preview without changes
|
||||
rsync_stop.sh --full-stop --dry-run # preview full stop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## user_scripts_stop.sh
|
||||
|
||||
### Process Identification
|
||||
|
||||
Scans `/proc/*/cmdline` for any process whose command line contains
|
||||
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
|
||||
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
|
||||
what the script is named or how it was launched.
|
||||
|
||||
Script names are extracted from the path for display: you see which scripts are
|
||||
being stopped, not just PIDs.
|
||||
|
||||
### Self-Exclusion
|
||||
|
||||
If this script is run via the User Scripts plugin, it would find its own PID in
|
||||
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
|
||||
from the kill list.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all User Script processes
|
||||
user_scripts_stop.sh --status # show running scripts with names and elapsed time
|
||||
user_scripts_stop.sh --dry-run # show what would be stopped
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## server_reboot.sh
|
||||
|
||||
### Full Shutdown Sequence
|
||||
|
||||
```
|
||||
1. Pre-flight checks (warn only — do not block):
|
||||
- rsync running? → warn, suggest rsync_stop.sh first
|
||||
- mover running? → warn, suggest mover_stop.sh first
|
||||
- Emby sessions? → warn (active streams will be interrupted)
|
||||
|
||||
2. Wall message to all logged-in terminal users
|
||||
|
||||
3. unRAID dashboard notification
|
||||
|
||||
4. Wait REBOOT_SLEEP seconds (default: 30)
|
||||
|
||||
5. Graceful VM shutdown:
|
||||
virsh shutdown <each VM> (ACPI signal — clean shutdown)
|
||||
Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond
|
||||
|
||||
6. /etc/rc.d/rc.libvirt stop (VM Manager)
|
||||
|
||||
7. /etc/rc.d/rc.docker stop (all containers stop)
|
||||
|
||||
8. sync (flush filesystem buffers to disk)
|
||||
|
||||
9. /sbin/reboot
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
REBOOT_SLEEP=30 # seconds between warning and shutdown sequence
|
||||
REBOOT_VM_WAIT=30 # seconds to wait for VMs to shut down gracefully
|
||||
```
|
||||
|
||||
### Recommended Pre-Reboot Sequence
|
||||
|
||||
For a clean reboot when services are active:
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # stop any active rsync (smart mode)
|
||||
mover_stop.sh # stop mover gracefully
|
||||
server_reboot.sh --status # check what's still running
|
||||
server_reboot.sh --reason="planned maintenance"
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
server_reboot.sh # reboot with 30s warning
|
||||
server_reboot.sh --dry-run # walk through without rebooting
|
||||
server_reboot.sh --status # show what would be affected
|
||||
server_reboot.sh --reason="disk work" # include reason in notification
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── System Watchdog ────────────────────────────────────────────────────────────
|
||||
SYS_WATCHDOG_INTERVAL=300 # seconds between check cycles
|
||||
SYS_WATCHDOG_STRIKES=3 # consecutive failures before reboot
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # rate limit window
|
||||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots in window
|
||||
SYS_WATCHDOG_OOM_LIMIT=3 # OOM kills/cycle for URGENT bypass
|
||||
|
||||
MEM_WARN_GB=10 # warn + notify
|
||||
MEM_SHUTDOWN_GB=6 # stop non-essential containers
|
||||
MEM_GB=4 # strike → reboot
|
||||
MEM_RECOVER_GB=30 # recovery threshold
|
||||
|
||||
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99 # Tier 1 trigger
|
||||
SYS_WATCHDOG_FD_CRITICAL_PCT=95 # Tier 1 trigger
|
||||
SYS_WATCHDOG_LOAD_MULTIPLIER=4 # Tier 3 — ×cpu_count
|
||||
SYS_WATCHDOG_CPU_TEMP=85 # Tier 3 — Celsius
|
||||
SYS_WATCHDOG_ZOMBIES=20 # Tier 3 — process count
|
||||
SYS_WATCHDOG_VAR_LOG_PCT=80 # Tier 3 — percent full
|
||||
SYS_WATCHDOG_TMP_PCT=85 # Tier 3 — percent full
|
||||
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=true
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=true
|
||||
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||||
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
|
||||
)
|
||||
SYS_WATCHDOG_REQUIRED_CONTAINERS=() # containers that must be running
|
||||
|
||||
# State files:
|
||||
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||||
SYS_WATCHDOG_REBOOT_LOG="/tmp/sys_watchdog_reboots.db"
|
||||
SYS_WATCHDOG_FAILED_FILE="/tmp/sys_watchdog_failed.db"
|
||||
SYS_WATCHDOG_OOM_FILE="/tmp/sys_watchdog_oom.db"
|
||||
|
||||
# ── Resource Watchdog ──────────────────────────────────────────────────────────
|
||||
RW_ENABLED=true
|
||||
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
|
||||
|
||||
RW_RAM_SOFT_GB=20
|
||||
RW_RAM_MEDIUM_GB=15
|
||||
RW_RAM_HARD_GB=10
|
||||
RW_RAM_RECOVER_GB=25
|
||||
|
||||
RW_LOAD_SOFT_MULTIPLIER=2.0
|
||||
RW_LOAD_MEDIUM_MULTIPLIER=3.0
|
||||
RW_RECOVER_CYCLES=3
|
||||
|
||||
RW_SABNZBD_ENABLED=true
|
||||
RW_SABNZBD_SPEED_SOFT="50M"
|
||||
RW_SABNZBD_SPEED_MEDIUM="10M"
|
||||
RW_QBIT_ENABLED=true
|
||||
RW_QBIT_DL_SOFT=51200 # KB/s
|
||||
RW_QBIT_DL_MEDIUM=10240
|
||||
|
||||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
|
||||
|
||||
# ── Per-Host (master_host*.conf) ───────────────────────────────────────────────
|
||||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
|
||||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
|
||||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||||
HOST1_QBIT_URL="http://localhost:8090"
|
||||
HOST1_QBIT_USERNAME="admin"
|
||||
HOST1_QBIT_PASSWORD="your-password"
|
||||
|
||||
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
WEBGUI_NGINX_WAIT=15
|
||||
WEBGUI_PHP_WAIT=10
|
||||
WEBGUI_EMHTTP_WAIT=30
|
||||
|
||||
# ── inotify Tuning ─────────────────────────────────────────────────────────────
|
||||
INOTIFY_MAX_INSTANCES=1024
|
||||
INOTIFY_MAX_WATCHES=1048576
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768
|
||||
|
||||
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
|
||||
PHP_MAX_CHILDREN=250
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
|
||||
# ── Syslog Filter ──────────────────────────────────────────────────────────────
|
||||
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
||||
|
||||
# ── Log Cleaner ────────────────────────────────────────────────────────────────
|
||||
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
|
||||
LOG_MIN_SIZE_MB=10
|
||||
LOG_DOCKER_MAX_MB=100
|
||||
|
||||
# ── Mover Stop ─────────────────────────────────────────────────────────────────
|
||||
MOVER_STOP_TIMEOUT=30
|
||||
|
||||
# ── Server Reboot ──────────────────────────────────────────────────────────────
|
||||
REBOOT_SLEEP=30
|
||||
REBOOT_VM_WAIT=30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### system_watchdog.sh Not Starting
|
||||
|
||||
```bash
|
||||
# Check if already running (acquire_lock prevents second instance):
|
||||
pgrep -a -f system_watchdog.sh
|
||||
|
||||
# Check state file permissions:
|
||||
ls -la /tmp/sys_watchdog_*.db
|
||||
|
||||
# Run with --log to see startup:
|
||||
system_watchdog.sh --log
|
||||
```
|
||||
|
||||
### system_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check reboot log:
|
||||
cat /tmp/sys_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check what condition triggered it:
|
||||
# Look in /var/log/syslog for "system_watchdog" near the reboot time
|
||||
grep "system_watchdog" /var/log/syslog | tail -20
|
||||
```
|
||||
|
||||
### resource_watchdog Paused Containers It Shouldn't Have
|
||||
|
||||
```bash
|
||||
# Check current state:
|
||||
resource_watchdog.sh --status
|
||||
|
||||
# Add the container to RW_CRITICAL_CONTAINERS in master.conf:
|
||||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
|
||||
|
||||
# Un-pause manually if needed:
|
||||
docker unpause MyContainer
|
||||
```
|
||||
|
||||
### rsync_stop Killed the Wrong Thing
|
||||
|
||||
If --full-stop killed an orchestrator you didn't intend to kill:
|
||||
|
||||
```bash
|
||||
# Next time use default mode (no --full-stop) to kill only rsync subprocess.
|
||||
# To verify what would be killed before running:
|
||||
rsync_stop.sh --status # shows running rsync and detected orchestrators
|
||||
rsync_stop.sh --dry-run # shows smart mode decision
|
||||
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
|
||||
```
|
||||
|
||||
### WebGUI Recovery After All Three Steps Failed
|
||||
|
||||
```bash
|
||||
# Check if processes are running:
|
||||
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
|
||||
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
|
||||
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
|
||||
|
||||
# Check recent nginx errors:
|
||||
cat /var/log/nginx/error.log | tail -20
|
||||
|
||||
# Check emhttp log:
|
||||
tail -20 /var/log/syslog | grep emhttp
|
||||
|
||||
# Last resort — reboot:
|
||||
server_reboot.sh --reason="WebGUI unrecoverable"
|
||||
```
|
||||
|
||||
### PHP-FPM Config Not Found After unRAID Update
|
||||
|
||||
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
|
||||
errors with "config file not found":
|
||||
|
||||
```bash
|
||||
# Find the new config path:
|
||||
find /etc -name "www.conf" 2>/dev/null
|
||||
|
||||
# Update PHP_CONF in master.conf:
|
||||
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
|
||||
|
||||
# Verify with:
|
||||
php_fpm_max_children.sh --status
|
||||
```
|
||||
@@ -1,807 +1,176 @@
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# 🖥️ UNRAID ESSENTIALS
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# ━━━━━ UNRAID ESSENTIALS ━━━━━
|
||||
|
||||
**System-level scripts that act on the unRAID server itself — not containers, not
|
||||
media, not monitoring.** Keeping the server stable under load, recovering a frozen
|
||||
WebGUI, tuning kernel limits, suppressing log noise, and handling graceful shutdowns
|
||||
and reboots with proper warning sequences.
|
||||
**System-level scripts that act on the unRAID server itself — not containers,
|
||||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||||
shutdowns with proper warning sequences.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Server Getting Into Unstable States With No Recovery Path**
|
||||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||||
write log files. The server is functionally dead but still technically running.
|
||||
|
||||
Fix: `system_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||||
threshold breaches. Last line of defense before a hard crash.
|
||||
|
||||
**WebGUI Freezing and Nobody Noticing**
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||||
container-level alert fires because this isn't a container problem — it's a web
|
||||
server problem. By the time someone notices it may have been broken for hours.
|
||||
|
||||
Fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||||
|
||||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||||
network interface for each one. Each interface generates multiple syslog entries.
|
||||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||||
|
||||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||||
starts. Applied at array start. Idempotent — silent when already correct.
|
||||
|
||||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||||
those workers saturate immediately. New requests queue behind active ones.
|
||||
|
||||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||||
|
||||
**inotify Exhaustion Producing Unexplained Failures**
|
||||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||||
and all containers share the same pool.
|
||||
|
||||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||||
|
||||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||||
array simultaneously. unRAID's deduplication layer is confused.
|
||||
|
||||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
```
|
||||
unRAID_Essentials/ ← acts on the server itself (this folder)
|
||||
Docker_Essentials/ ← acts on containers
|
||||
Media/ ← acts on the library
|
||||
Monitors/ ← observes and reports
|
||||
```
|
||||
|
||||
> **The escalation chain matters here.** Docker_Essentials handles container-level
|
||||
> problems. unRAID_Essentials handles server-level problems. The watchdogs are
|
||||
> designed to work together — docker_watchdog.sh heals containers first,
|
||||
> system_watchdog.sh reboots only when healing has failed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Server Getting Into Unstable States With No Recovery Path
|
||||
|
||||
A container has a severe memory leak. RAM drops to 2GB. The system starts swapping.
|
||||
Everything slows down. Docker watchdog tries to restart the container — but Docker
|
||||
itself is barely responding. The restart hangs. The watchdog is stuck. Nothing is
|
||||
getting better. The server needs a reboot, but nothing in the ecosystem is authorised
|
||||
to call one.
|
||||
|
||||
Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The WebGUI
|
||||
shows nothing useful. The server is functionally dead but still technically running.
|
||||
Again — needs a reboot, nothing calls one.
|
||||
|
||||
The fix: `system_watchdog.sh` — the last line of defense. Three-tier response:
|
||||
immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, and a
|
||||
strike system for sustained threshold breaches. When everything else has failed,
|
||||
system_watchdog reboots cleanly before a hard crash happens.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 WebGUI Freezing and Nobody Noticing
|
||||
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state. Or PHP-FPM workers are
|
||||
saturated and new requests are queueing indefinitely. Or emhttp itself has frozen.
|
||||
From a user perspective: dashboard doesn't load, settings don't save, containers
|
||||
can't be started or stopped via the UI.
|
||||
|
||||
Nothing in the container stack alerts on a frozen WebGUI — it's not a container
|
||||
problem, it's a web server problem. The only way to know is if someone tries to use
|
||||
the UI and notices. By which point it may have been broken for hours.
|
||||
|
||||
The fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Notifies on any restart so you know it happened.
|
||||
Silent when healthy — 144 runs per day with no output is the correct behaviour.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 50+ Containers Starting and Filling Syslog With Veth Noise
|
||||
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual network
|
||||
interface for each one. Each creation generates multiple syslog entries. In the first
|
||||
few minutes after array start the syslog is buried under hundreds of lines of:
|
||||
|
||||
```
|
||||
kernel: veth2a3b4c5: renamed from eth0
|
||||
kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||||
kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||||
```
|
||||
|
||||
Real events — a failed mount, a permission error, a service that didn't start —
|
||||
are invisible in this noise. And on a busy server that restarts containers regularly,
|
||||
this noise continues throughout the day.
|
||||
|
||||
The fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any
|
||||
container starts. Applied at array start, idempotent, silent when already correct.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 WebGUI Queueing Requests Under Load Without Explanation
|
||||
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Saving settings seems to
|
||||
hang. Nothing in the logs explains it. Container starts from the UI timeout. The
|
||||
server itself is not under load — CPU is fine, RAM is fine.
|
||||
|
||||
The cause: PHP-FPM's `pm.max_children` defaults to 4-8 workers. On a server with
|
||||
multiple users, active plugins, automated tools polling the API, and 50+ containers
|
||||
all potentially hitting the WebGUI simultaneously, those 4-8 workers saturate
|
||||
immediately. New requests queue behind active ones. Everything feels slow.
|
||||
|
||||
The fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~40MB = ~10GB worst case. On 128GB this is trivially small. The
|
||||
WebGUI becomes responsive immediately. Idempotent — silent when already correct.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 inotify Exhaustion Producing Unexplained Failures
|
||||
|
||||
Already documented in README-Monitors.md (system_tuning_monitor.sh section). Short
|
||||
version: when inotify limits are exhausted, containers silently stop receiving file
|
||||
system events. Downloads complete but arrs don't detect them. The kernel hits the
|
||||
limit and new watches fail silently. VSCode shows "unable to watch for file changes"
|
||||
and misses edits.
|
||||
|
||||
The fix: `inotify_tuning.sh` — raises all three inotify limits at array start.
|
||||
1M watches (raised from 512K — Code-Server with node_modules needs this), 1024
|
||||
instances, 32768 queued events. The startup race note: if Code-Server starts before
|
||||
this runs, it inherits old limits. Restart Code-Server if the VSCode error appears
|
||||
after limits are applied.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Mover Getting Killed Mid-Transfer Leaving Files Inconsistent
|
||||
|
||||
The mover is running — moving a large batch of files from cache to array. Someone
|
||||
clicks reboot from the UI. Or a script kills the mover process directly. The mover
|
||||
stops mid-file. The file exists partially on both cache and array simultaneously.
|
||||
unRAID's deduplication layer is confused. The file is inaccessible.
|
||||
|
||||
The fix: `mover_stop.sh` — warns logged-in users via wall message, waits the
|
||||
configured timeout, then sends SIGTERM (graceful) and verifies. The mover gets to
|
||||
finish its current file operation before stopping. SIGKILL is a last resort with a
|
||||
warning that partial files may exist.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Four distinct server-level roles:
|
||||
|
||||
```
|
||||
🛡️ Last-resort stability system_watchdog.sh — reboots before crash
|
||||
🌐 WebGUI availability webgui_restart.sh — recovers frozen UI
|
||||
⚙️ Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
🔇 Log hygiene docker_syslog_filter.sh — suppress veth noise
|
||||
clear_logs.sh — weekly log trimming
|
||||
🔄 Graceful operations mover_stop.sh — clean mover stop
|
||||
server_reboot.sh — clean reboot with warning
|
||||
user_scripts_stop.sh — stop running scripts
|
||||
Last-resort stability system_watchdog.sh — reboots before crash
|
||||
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
|
||||
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
|
||||
Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||||
clear_logs.sh — weekly log trimming
|
||||
Graceful operations mover_stop.sh — clean mover stop
|
||||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||||
user_scripts_stop.sh — stop running User Scripts
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
| Script | Purpose | When |
|
||||
|--------|---------|------|
|
||||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop |
|
||||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | Every 10 minutes |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM worker count | At array start |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth log noise | At array start |
|
||||
| `clear_logs.sh` | Size-threshold weekly log cleanup | Weekly via maintenance window |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual or called by system_watchdog |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual or called by server_reboot |
|
||||
```
|
||||
Orchestrators/
|
||||
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||||
─────────────────────────► docker_syslog_filter.sh (second)
|
||||
─────────────────────────► php_fpm_max_children.sh
|
||||
─────────────────────────► system_watchdog.sh (background loop)
|
||||
|
||||
---
|
||||
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🛡️ system_watchdog.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
|
||||
The last line of defense. Reboots the system cleanly before it crashes uncleanly.
|
||||
Three-tier response system — critical failures bypass everything and reboot immediately,
|
||||
OOM-confirmed crises bypass the strike system, sustained threshold breaches use strikes.
|
||||
Runs continuously as a background process started by `array_start.sh`.
|
||||
|
||||
> Full architecture documentation in `README-Docker_Essentials.md` under
|
||||
> "Relationship to System Watchdog". This section covers the system_watchdog itself.
|
||||
|
||||
```bash
|
||||
# Started by: array_start.sh (continuous background process)
|
||||
# Interval: SYSTEM_WATCHDOG_INTERVAL=300 (5 minutes)
|
||||
Docker_Essentials/
|
||||
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
|
||||
(mem_shutdown_active flag)
|
||||
```
|
||||
|
||||
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
|
||||
designed to work together — docker_watchdog heals containers first,
|
||||
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
|
||||
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
|
||||
prevent docker_watchdog from restarting containers that resource_watchdog just
|
||||
stopped to free RAM.
|
||||
|
||||
---
|
||||
|
||||
### ── Three-Tier Response System ──────────────────────────────────────────────
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# TIER 1 — CRITICAL (bypass ALL strikes, reboot immediately)
|
||||
# These failures are acute — the system is not recoverable by waiting.
|
||||
# Single detection = immediate reboot. No confirmation window.
|
||||
#
|
||||
# Docker daemon unresponsive:
|
||||
# Attempt /etc/rc.d/rc.docker restart first.
|
||||
# Wait 15 seconds. Verify daemon responding.
|
||||
# If still hung → CRITICAL reboot.
|
||||
# A hung daemon cannot be healed — every subsequent docker command hangs.
|
||||
#
|
||||
# rootfs at ROOTFS_CRITICAL_PCT (99%+):
|
||||
# Writes are failing. SSH may stop. Logs can't be written.
|
||||
# Nothing can be fixed from this state without a reboot.
|
||||
#
|
||||
# Kernel BUG/Oops in dmesg:
|
||||
# Kernel running with corrupted state.
|
||||
# Delta-based: new oops since last cycle → reboot.
|
||||
#
|
||||
# File descriptor exhaustion (FD_CRITICAL_PCT=95%):
|
||||
# New connections failing. Docker can't spawn processes. SSH drops.
|
||||
#
|
||||
# /boot read-only:
|
||||
# State files and config writes silently failing.
|
||||
# Write test on /boot every cycle.
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
|
||||
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
|
||||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||
|
||||
# TIER 2 — URGENT (bypass strikes when OOM confirms crisis)
|
||||
# RAM < MEM_GB (4GB) AND OOM kills >= OOM_LIMIT (3) in this cycle
|
||||
# Without OOM confirmation → standard strike system applies.
|
||||
# Rationale: 1-2 OOM kills = docker_watchdog.sh handles it.
|
||||
# 3+ kills while RAM critical = system dying faster than watchdogs heal.
|
||||
# OOM victims from dmesg included in reboot message (diagnostic context).
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
# TIER 3 — STANDARD (strike system — N consecutive failures → reboot)
|
||||
# Everything else: RAM tiers, load, CPU temp, zombies, /var/log,
|
||||
# /tmp, containers, NIC, mdstat, sshd
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||||
├─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
└─ system_watchdog.sh ← starts background loop
|
||||
|
||||
---
|
||||
|
||||
### ── RAM Tiers ────────────────────────────────────────────────────────────────
|
||||
Every minute (watchdog_orchestrator.sh):
|
||||
└─ resource_watchdog.sh
|
||||
Level 1 (soft): throttle SABnzbd + qBit download speeds
|
||||
Level 2 (medium): further throttle + docker pause non-essential containers
|
||||
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
|
||||
↓
|
||||
docker_watchdog.sh reads mem_shutdown_active — defers restarts
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Three-level graduated response — not a single threshold.
|
||||
#
|
||||
SYS_WATCHDOG_MEM_WARN_GB=10 # warn + notify once — informational only
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_GB=6 # stop non-essential containers, wait for recovery
|
||||
SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (or bypass if OOM confirms)
|
||||
SYS_WATCHDOG_MEM_RECOVER_GB=30 # RAM must reach this before restarting containers
|
||||
#
|
||||
# Containers excluded from RAM emergency shutdown — these stay running:
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||||
"NginxProxyManager" # external access — stop this and users lose everything
|
||||
"Authelia" # auth — without this nothing is accessible
|
||||
"Mariadb" # Authelia dependency
|
||||
"Redis" # Authelia dependency
|
||||
"Emby" # media server — Live TV buffering
|
||||
"Dispatcharr" # Live TV scheduler — loses state if stopped
|
||||
)
|
||||
#
|
||||
# Coordination with docker_watchdog.sh:
|
||||
# system_watchdog writes mem_shutdown_active=true to SYS_WATCHDOG_STATE_FILE.
|
||||
# docker_watchdog reads this flag and defers ALL container restart logic.
|
||||
# Without this: both watchdogs fight — system_watchdog stops containers,
|
||||
# docker_watchdog restarts them, RAM never recovers.
|
||||
# With this: docker_watchdog stands down until mem_shutdown_active clears.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
Every 10 minutes (User Scripts):
|
||||
└─ webgui_restart.sh
|
||||
WebGUI OK → silent exit
|
||||
Not responding:
|
||||
Step 1: restart nginx → recheck
|
||||
Step 2: restart php-fpm → recheck
|
||||
Step 3: restart emhttp → recheck
|
||||
All failed → notify, exit 1
|
||||
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
System logs: clear if > LOG_MIN_SIZE_MB
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
|
||||
Manual operations:
|
||||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Per-Host Check Toggles ───────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf (all checks in master_host*.conf — not master.conf)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Each check is independently toggleable per server.
|
||||
# HOST1 and HOST2 may have different hardware and different workloads.
|
||||
# detect_hosts() aliases HOST*_SYS_WATCHDOG_CHECK_* → SYS_WATCHDOG_CHECK_*
|
||||
#
|
||||
# Tier 1 — Critical (bypass strikes):
|
||||
HOST1_SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_ROOTFS=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_KERNEL_OOPS=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_FD=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_BOOT=true
|
||||
|
||||
# Tier 2 — Urgent (OOM bypass):
|
||||
HOST1_SYS_WATCHDOG_CHECK_OOM=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_RAM=true
|
||||
|
||||
# Tier 3 — Standard (strike system):
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOG=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_ARC=true # HOST1 runs ZFS — enable
|
||||
HOST1_SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOAD=false # disabled — transcoding causes normal spikes
|
||||
HOST1_SYS_WATCHDOG_CHECK_ZOMBIES=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_CONTAINERS=true # monitors docker_watchdog persistent skip list
|
||||
HOST1_SYS_WATCHDOG_CHECK_TMP=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_MDSTAT=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_NETWORK=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_SSHD=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false # disabled — may false positive during encoding
|
||||
HOST1_SYS_WATCHDOG_NIC="eth0" # verify: ip link show | grep "^[0-9]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Abort Conditions ─────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Conditions that prevent a reboot even when a threshold is hit.
|
||||
# CRITICAL tier bypasses these — truly critical conditions reboot regardless.
|
||||
#
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # rebooting with bad pool risks data loss
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity is better than crashing
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting mover is better than crashing
|
||||
#
|
||||
# Philosophy: a graceful reboot before a crash is always better than a hard crash.
|
||||
# The abort conditions protect against the cases where a reboot itself causes harm
|
||||
# (data loss from bad ZFS pool). Parity and mover can be restarted after reboot.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Reboot Loop Protection ───────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# If the server keeps rebooting, something is wrong that rebooting isn't fixing.
|
||||
# After REBOOT_LIMIT reboots in REBOOT_WINDOW_HRS → shutdown instead.
|
||||
# Shutdown prevents: hardware damage, filesystem corruption from repeated reboots,
|
||||
# infinite loop that never lets you investigate.
|
||||
# State file: /boot/config/system_watchdog_reboots.db — survives reboots.
|
||||
#
|
||||
SYS_WATCHDOG_REBOOT_LIMIT=3 # reboots before shutdown instead
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12 # rolling window in hours
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── State File Heartbeat ─────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# system_watchdog writes watchdog_cycle=N to SYS_WATCHDOG_STATE_FILE every cycle.
|
||||
# This keeps the file's modification time current.
|
||||
#
|
||||
# docker_watchdog.sh uses the state file mtime as a stale guard — if the file
|
||||
# is more than 2 hours old while mem_shutdown_active=true is set, system_watchdog
|
||||
# may have stopped running. docker_watchdog resumes normal operation rather than
|
||||
# being silenced indefinitely by a stale flag.
|
||||
#
|
||||
# Without this heartbeat: if all standard checks pass and no state writes happen
|
||||
# (e.g. CHECK_KERNEL_OOPS=false AND CHECK_MDSTAT=false), the file mtime could go
|
||||
# stale even with the watchdog running.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # normal (continuous — started by array_start.sh)
|
||||
system_watchdog.sh --dry-run # trigger detection without rebooting
|
||||
system_watchdog.sh --status # show all tiers, thresholds, active strikes, RAM state
|
||||
system_watchdog.sh --log # verbose per-cycle output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🌐 webgui_restart.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Monitors the unRAID WebGUI availability and recovers it automatically when unresponsive.
|
||||
Three-step escalating strategy — lightest fix first, heaviest last. Silent when healthy.
|
||||
|
||||
```bash
|
||||
# Scheduled: */10 * * * * (every 10 minutes)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Three-Step Escalation ────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 1 — nginx restart
|
||||
# Lightest fix — handles most WebGUI failures.
|
||||
# nginx is the web server layer. Crash, worker stuck, connection timeout.
|
||||
# Wait WEBGUI_NGINX_WAIT seconds → curl recheck.
|
||||
#
|
||||
# Step 2 — php-fpm restart
|
||||
# Added because WebGUI can appear frozen due to PHP worker exhaustion.
|
||||
# pm.max_children workers all occupied → new requests queue → dashboard hangs.
|
||||
# php-fpm restart far less disruptive than emhttp.
|
||||
# Wait WEBGUI_PHP_WAIT seconds → curl recheck.
|
||||
# Note: system_tuning_monitor.sh tracks worker saturation over time.
|
||||
#
|
||||
# Step 3 — emhttp restart
|
||||
# Heaviest fix. emhttp is the core unRAID management daemon.
|
||||
# Array, Docker, shares continue running — only WebGUI management restarts.
|
||||
# Takes longer to recover — WEBGUI_EMHTTP_WAIT gives it time.
|
||||
# Wait WEBGUI_EMHTTP_WAIT seconds → curl recheck.
|
||||
#
|
||||
# If all three fail → notify warning — manual intervention needed.
|
||||
# Guidance: pgrep -x nginx emhttp | check journalctl | consider server_reboot.sh
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Configuration ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WEBGUI_URL="http://localhost" # adjust if non-standard port
|
||||
WEBGUI_TIMEOUT=5 # seconds before curl times out
|
||||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart — takes longer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
webgui_restart.sh # check once — silent if healthy, escalates if not
|
||||
webgui_restart.sh --dry-run # walk through escalation without restarting anything
|
||||
webgui_restart.sh --status # show WebGUI state + nginx/php-fpm/emhttp process state
|
||||
webgui_restart.sh --log # verbose — show each escalation step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 📡 inotify_tuning.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Raises Linux inotify kernel limits at array start. Idempotent — completely silent
|
||||
when values are already correct.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh — FIRST in ARRAY_START_SCRIPTS)
|
||||
# Must run before containers start — containers inherit limits at startup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Three Limits ─────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
INOTIFY_MAX_INSTANCES=1024 # default: 128 — max inotify instances per user
|
||||
# 1024 handles ~20-30 containers watching files
|
||||
|
||||
INOTIFY_MAX_WATCHES=1048576 # default: 8192 — SHARED budget across ALL users/containers
|
||||
# Was 512K — raised to 1M (1048576)
|
||||
# VSCode/Code-Server alone needs ~50K-200K for large workspaces
|
||||
# with node_modules. All arr containers + Emby + VSCode share
|
||||
# this budget. 1M safe on 128GB RAM (~128MB kernel memory)
|
||||
# If VSCode shows "unable to watch for file changes" → too low
|
||||
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768 # default: 16384 — events buffered before dropping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Startup Race Condition ───────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# inotify limits are kernel-wide — they take effect immediately on sysctl write.
|
||||
# But containers that have ALREADY started inherit the OLD limits at startup.
|
||||
# Those containers keep their inherited (low) limits until restarted.
|
||||
#
|
||||
# This is why inotify_tuning.sh must be FIRST in ARRAY_START_SCRIPTS — before
|
||||
# any container starts. If Code-Server starts before limits are raised:
|
||||
# → Code-Server inherits old 8192 watch limit
|
||||
# → VSCode shows "unable to watch for file changes"
|
||||
# → Fix: docker restart Code-Server (picks up current kernel limits on start)
|
||||
#
|
||||
# The script warns if it changed any values:
|
||||
# "If Code-Server is running: docker restart Code-Server"
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
inotify_tuning.sh # normal run (idempotent — silent when correct)
|
||||
inotify_tuning.sh --dry-run # show what would change without changing
|
||||
inotify_tuning.sh --status # current values vs targets + top inotify consumers
|
||||
inotify_tuning.sh --log # verbose — show each sysctl write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ⚙️ php_fpm_max_children.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Raises PHP-FPM `pm.max_children` at array start to prevent WebGUI queueing under
|
||||
load. Idempotent — completely silent when already correct. No PHP-FPM restart unless
|
||||
value actually changed.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Why 250 Workers ──────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
PHP_MAX_CHILDREN=250 # default: 4-8 — far too low for a busy server
|
||||
# Each worker: ~2MB resident memory at idle
|
||||
# 250 workers × 2MB = ~500MB — trivial on 128GB
|
||||
# Worst case (all active): ~250 × 40MB = ~10GB
|
||||
# In practice: rarely all active simultaneously
|
||||
# On 64GB (HOST2): still appropriate — 250 × 40MB
|
||||
# = 10GB worst case = 15% of RAM, acceptable
|
||||
#
|
||||
# Without this fix:
|
||||
# 5 users hit the WebGUI simultaneously → 8 workers exhausted → 5 more queue
|
||||
# Each queued request waits for a worker to free → 5-10 second response times
|
||||
# Looks like a slow server — it's just a queue
|
||||
#
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf" # path may change with PHP version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
php_fpm_max_children.sh # normal run (idempotent)
|
||||
php_fpm_max_children.sh --dry-run # show what would change
|
||||
php_fpm_max_children.sh --status # show current value vs target + worker count
|
||||
php_fpm_max_children.sh --log # verbose — show config write + restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔇 docker_syslog_filter.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Creates an rsyslog drop rule for Docker veth/docker0 interface noise. Idempotent —
|
||||
silent when filter already correct, only writes + restarts rsyslog when something changed.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh — before containers start)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What Gets Suppressed ─────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# These kernel messages are generated on every container start and stop:
|
||||
#
|
||||
# kernel: veth2a3b4c5: renamed from eth0
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered disabled state
|
||||
#
|
||||
# 50+ containers at array start = 200-400 lines of this in the first minute.
|
||||
# Containers restart throughout the day = continuous noise.
|
||||
# Real events buried and invisible in syslog.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Idempotent Design ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Runs at every array start but only changes something when needed:
|
||||
# Filter file exists and content is correct → exit 0 silently
|
||||
# Filter file missing or content changed → write + restart rsyslog
|
||||
#
|
||||
# Expected content compared exactly — single source of truth:
|
||||
EXPECTED_FILTER='if ($msg contains "veth" or $msg contains "docker0") then {
|
||||
stop
|
||||
}'
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
docker_syslog_filter.sh # normal run (idempotent — silent when correct)
|
||||
docker_syslog_filter.sh --dry-run # show what would be written without writing
|
||||
docker_syslog_filter.sh --status # show current filter file + rsyslog state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🗑️ clear_logs.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Clears system logs and Docker container logs using size thresholds. Only clears logs
|
||||
large enough to be worth clearing — preserves recent diagnostic context on small logs.
|
||||
|
||||
```bash
|
||||
# Called by: weekly_sync_maintenance.sh via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Size Threshold Approach ──────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Why thresholds instead of clearing everything:
|
||||
# A 2MB syslog contains useful recent history — not worth clearing.
|
||||
# A 500MB Docker log is consuming rootfs and contains mostly noise — clear it.
|
||||
# Blind truncation destroys diagnostic context for no benefit.
|
||||
#
|
||||
LOG_MIN_SIZE_MB=10 # skip system log if under this size — keep history
|
||||
LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this size
|
||||
# Active containers (Emby, SABnzbd) grow fastest
|
||||
# 100MB × 30 containers = 3GB before any clearing kicks in
|
||||
|
||||
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
|
||||
#
|
||||
# Truncation not logrotate:
|
||||
# unRAID writes to tmpfs (/var/log) — logrotate's compress + archive approach
|
||||
# would consume more tmpfs space, not less.
|
||||
# : > file keeps the file descriptor valid while emptying content.
|
||||
# Safe for running services (syslogd continues writing to the same fd).
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
clear_logs.sh # normal run — silent if all logs under threshold
|
||||
clear_logs.sh --dry-run # show what would be cleared and sizes
|
||||
clear_logs.sh --status # show current log sizes vs thresholds
|
||||
clear_logs.sh --log # verbose — show each file evaluated
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ⏹️ mover_stop.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops the unRAID mover cleanly — wall warning, configurable timeout, SIGTERM → verify
|
||||
→ SIGKILL sequence. Safe to run when mover is not running — exits cleanly with a log.
|
||||
|
||||
---
|
||||
|
||||
### ── Stop Sequence ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Check if mover is running — exit cleanly if not
|
||||
# 2. Wall message: "HOST1 (unRAID-Gmer4Lfe) — mover stopping in 30s"
|
||||
# MY_ID included — on shared terminal it's clear which server
|
||||
# 3. Wait MOVER_STOP_TIMEOUT seconds
|
||||
# 4. SIGTERM — allows mover to finish its current file before stopping
|
||||
# No partial files — the mover completes what it's working on
|
||||
# 5. Wait 5 seconds — verify if stopped
|
||||
# 6. If still running → SIGKILL (forced)
|
||||
# Warning: partial files possible — same as a hard crash
|
||||
# 7. Final verify — error if still running
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
mover_stop.sh # stop mover with configured timeout
|
||||
mover_stop.sh --dry-run # show what would happen
|
||||
mover_stop.sh --status # show mover state (running, PID, start time)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔁 server_reboot.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Graceful reboot with pre-flight warnings, wall message, unRAID notification, VM
|
||||
graceful shutdown, then Docker and services stop, sync, reboot.
|
||||
|
||||
> Full documentation in `README-Tools.md` — `server_reboot.sh` section. This is a
|
||||
> quick reference.
|
||||
|
||||
---
|
||||
|
||||
### ── Shutdown Sequence ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Pre-flight warnings (warn not block):
|
||||
# rsync running, mover running, active Emby sessions
|
||||
# Warnings show in summary — you chose to reboot, these are for context
|
||||
# 2. Wall message + unRAID notification — MY_ID included
|
||||
# 3. Wait REBOOT_SLEEP seconds (default 30)
|
||||
# 4. virsh shutdown each VM → wait REBOOT_VM_WAIT seconds for graceful exit
|
||||
# 5. Stop libvirt (VM Manager)
|
||||
# 6. Stop Docker service
|
||||
# 7. sync — flush filesystem buffers
|
||||
# 8. /sbin/reboot
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
server_reboot.sh # reboot with 30s warning
|
||||
server_reboot.sh --dry-run # full sequence walkthrough without rebooting
|
||||
server_reboot.sh --status # show running processes that would be affected
|
||||
server_reboot.sh --reason="maintenance" # include reason in wall + notification
|
||||
server_reboot.sh --log # verbose per-step output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🛑 user_scripts_stop.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops all running User Script processes. Identifies by `/tmp/user.scripts` path
|
||||
signature. Shows script names not PIDs. SIGTERM → verify → SIGKILL with self-exclusion.
|
||||
|
||||
> Full documentation in `README-Tools.md` — `user_scripts_stop.sh` section.
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all — SIGTERM → SIGKILL if needed
|
||||
user_scripts_stop.sh --dry-run # show which scripts would be stopped, by name
|
||||
user_scripts_stop.sh --status # show running scripts with PID and runtime
|
||||
user_scripts_stop.sh --log # verbose per-process output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ STARTUP SEQUENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# master.conf — ARRAY_START_SCRIPTS (order matters)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
ARRAY_START_SCRIPTS=(
|
||||
# ── One-shot — run and exit ───────────────────────────────────────────────
|
||||
"unRAID_Essentials/inotify_tuning.sh" # FIRST — raise limits before
|
||||
# containers inherit old values
|
||||
"unRAID_Essentials/docker_syslog_filter.sh" # SECOND — before containers
|
||||
# create veth interfaces
|
||||
"unRAID_Essentials/php_fpm_max_children.sh" # before WebGUI serves requests
|
||||
"Transcodes/ramdisk_setup.sh" # before Emby starts transcoding
|
||||
"Docker_Essentials/docker_network_connect.sh" # before watchdogs check states
|
||||
|
||||
# ── Continuous — run until array stops ───────────────────────────────────
|
||||
"unRAID_Essentials/system_watchdog.sh" # before docker_watchdog —
|
||||
# writes state file docker_watchdog reads
|
||||
"Docker_Essentials/docker_watchdog.sh" # before failover — containers
|
||||
# must be healthy for failover decisions
|
||||
"Failover/failover.sh" # last — needs everything stable
|
||||
)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# array_start.sh is the ONLY "At Startup of Array" entry in User Scripts.
|
||||
# It launches everything above in order.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FULL SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# At Startup of Array — via array_start.sh:
|
||||
# inotify_tuning.sh
|
||||
# docker_syslog_filter.sh
|
||||
# php_fpm_max_children.sh
|
||||
# (ramdisk_setup.sh — in Transcodes/)
|
||||
# system_watchdog.sh (continuous)
|
||||
|
||||
# Every 10 minutes:
|
||||
*/10 * * * * webgui_restart.sh # silent when healthy — escalates when not
|
||||
|
||||
# Weekly — via weekly_sync_maintenance.sh:
|
||||
# clear_logs.sh # Sunday 2:30am via WEEKLY_MAINTENANCE_SCRIPTS
|
||||
|
||||
# Manual:
|
||||
# mover_stop.sh — before array ops that need mover stopped
|
||||
# server_reboot.sh — planned maintenance reboots
|
||||
# user_scripts_stop.sh — emergency script stop or pre-reboot cleanup
|
||||
```
|
||||
@@ -2,51 +2,81 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Clear Logs =================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Clears system and Docker container logs to prevent rootfs fill over time.
|
||||
# Runs weekly via WEEKLY_MAINTENANCE_SCRIPTS — Sunday 2:30am.
|
||||
# Uses size thresholds — only clears logs that have grown large enough to matter.
|
||||
# Called weekly via WEEKLY_MAINTENANCE_SCRIPTS. Uses size thresholds — only
|
||||
# clears logs large enough to be worth clearing. Small logs are left intact,
|
||||
# preserving recent diagnostic context.
|
||||
#
|
||||
# ── WHAT IT CLEARS ────────────────────────────────────────────────────────────────────────────
|
||||
# System logs — LOG_FILES from master.conf (/var/log/syslog, messages, dmesg)
|
||||
# Cleared if size exceeds LOG_MIN_SIZE_MB
|
||||
# These grow continuously — weekly clearing keeps rootfs healthy
|
||||
# System logs (LOG_FILES): cleared if size exceeds LOG_MIN_SIZE_MB.
|
||||
# Docker logs (/var/lib/docker/containers/**/*-json.log): cleared only if the
|
||||
# individual container log exceeds LOG_DOCKER_MAX_MB. Active containers (Emby,
|
||||
# SABnzbd) grow fastest — inactive containers typically remain small.
|
||||
#
|
||||
# Docker logs — /var/lib/docker/containers/**/*-json.log
|
||||
# Cleared only if individual container log exceeds LOG_DOCKER_MAX_MB
|
||||
# Active containers (Emby, SABnzbd) grow fastest — 100MB+ easily
|
||||
# Inactive containers not cleared — their logs are typically small
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SIZE THRESHOLD APPROACH ───────────────────────────────────────────────────────────────────
|
||||
# Truncating everything blindly destroys useful diagnostic context.
|
||||
# A 2MB log is not worth clearing — it contains useful recent history.
|
||||
# A 500MB log is consuming rootfs and contains mostly noise — clear it.
|
||||
# Size Thresholds, Not Blind Truncation
|
||||
# A 2MB syslog contains useful recent diagnostic history — not worth clearing.
|
||||
# A 500MB Docker log is consuming rootfs and contains mostly noise — clear it.
|
||||
# Blind truncation destroys diagnostic context for no benefit.
|
||||
#
|
||||
# LOG_MIN_SIZE_MB — system logs under this size are left alone
|
||||
# LOG_DOCKER_MAX_MB — Docker logs under this size are left alone
|
||||
# Truncation, Not Logrotate
|
||||
# unRAID writes logs to tmpfs (/var/log). Logrotate's compress + archive approach
|
||||
# would consume more tmpfs space, not less. Truncation (`: > file`) keeps the
|
||||
# file descriptor open and valid while emptying content — syslogd continues
|
||||
# writing to the same fd without interruption.
|
||||
#
|
||||
# ── WHY NOT LOGROTATE ─────────────────────────────────────────────────────────────────────────
|
||||
# unRAID writes to tmpfs (/var/log) — logrotate's compress + archive approach
|
||||
# would consume even more tmpfs space. Truncation (: > file) keeps the file
|
||||
# descriptor open and valid while emptying content — safe for running services.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent runs corrupting logs
|
||||
# Root check — truncating system logs requires root
|
||||
# Size thresholds — only clears logs that have grown large enough
|
||||
# Byte tracking — reports MB freed for weekly digest
|
||||
# validate_unraid — notify validated before use
|
||||
# Silent on clean — small logs = nothing to clear = no output ✅
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs from corrupting logs.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# LOG_FILES — system log paths to check and clear
|
||||
# LOG_MIN_SIZE_MB — minimum system log size before clearing (default 10MB)
|
||||
# LOG_DOCKER_MAX_MB — clear Docker log only if above this size (default 100MB)
|
||||
# Root Required
|
||||
# Truncating system logs requires root.
|
||||
#
|
||||
# Size Thresholds
|
||||
# Each file checked against its threshold before clearing.
|
||||
#
|
||||
# Silent When Clean
|
||||
# All logs below threshold = no visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# LOG_FILES
|
||||
# System log paths to check and clear. (default: /var/log/syslog /var/log/messages /var/log/dmesg)
|
||||
#
|
||||
# LOG_MIN_SIZE_MB
|
||||
# Skip system log if under this size — keep recent history. (default: 10)
|
||||
#
|
||||
# LOG_DOCKER_MAX_MB
|
||||
# Clear Docker container log only if over this size. (default: 100)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# clear_logs.sh
|
||||
# Check all configured logs. Clear those above threshold. Silent when all are small.
|
||||
#
|
||||
# clear_logs.sh --dry-run
|
||||
# Show which logs would be cleared and their current sizes. No clearing.
|
||||
#
|
||||
# clear_logs.sh --status
|
||||
# Show current log sizes vs thresholds.
|
||||
#
|
||||
# clear_logs.sh --log
|
||||
# Verbose output — show each file evaluated, its size, and action taken.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# clear_logs.sh — normal run (threshold-based clearing)
|
||||
# clear_logs.sh --dry-run — show what would be cleared and sizes
|
||||
# clear_logs.sh --status — show current log sizes
|
||||
# clear_logs.sh --log — verbose output per file
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,49 +2,76 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Docker Syslog Filter ===========================================
|
||||
# ==============================================================================================
|
||||
# Suppresses noisy Docker veth/docker0 interface messages from syslog.
|
||||
# Run once at array start via ARRAY_START_SCRIPTS in master.conf.
|
||||
# Idempotent — completely silent when filter is already correct.
|
||||
#
|
||||
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
|
||||
# Every time Docker creates or destroys a container network interface it logs messages like:
|
||||
# kernel: veth2a3b4c5: renamed from eth0
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Suppresses Docker veth/docker0 network interface messages from syslog. Run
|
||||
# once at array start via ARRAY_START_SCRIPTS before any containers start.
|
||||
# Idempotent — completely silent when the filter is already in place.
|
||||
#
|
||||
# On a busy server creating and restarting many containers these fill syslog rapidly —
|
||||
# hundreds of entries per minute on container restarts, completely masking real events.
|
||||
# The filter tells rsyslog to drop these before they reach the log file.
|
||||
# Every container start/stop generates kernel messages like:
|
||||
# "veth2a3b4c5: renamed from eth0"
|
||||
# "docker0: port 1(veth...) entered forwarding state"
|
||||
# 50+ containers at array start = 200–400 lines of noise in the first minute,
|
||||
# completely masking real events.
|
||||
#
|
||||
# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
|
||||
# Creates /etc/rsyslog.d/ignore-docker-veth.conf (FILTER_FILE in master.conf).
|
||||
# rsyslog processes .conf files in /etc/rsyslog.d/ automatically on startup.
|
||||
# Filter uses rsyslog's RainerScript to match messages containing "veth" or "docker0"
|
||||
# and calls stop — the message is dropped before reaching any output target.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── IDEMPOTENT DESIGN ─────────────────────────────────────────────────────────────────────────
|
||||
# On every array start: checks if filter file already exists with correct content.
|
||||
# If already correct → completely silent — no rsyslog restart, no output.
|
||||
# Only writes + restarts rsyslog if filter is missing or content has changed.
|
||||
# This prevents unnecessary rsyslog restarts on every boot.
|
||||
# Idempotent Content Check
|
||||
# Checks whether the filter file already exists with exactly the expected content.
|
||||
# If already correct → silent exit, no rsyslog restart. Only writes and restarts
|
||||
# rsyslog when the filter is missing or content has changed. Running at every boot
|
||||
# without this would cause an unnecessary rsyslog restart on every array start.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — writing to /etc/rsyslog.d/ requires root
|
||||
# acquire_lock — prevents concurrent runs at array start
|
||||
# Idempotent check — only restarts rsyslog when filter actually changed
|
||||
# Directory creation — mkdir -p /etc/rsyslog.d/ before writing
|
||||
# rsyslog verify — checks rsyslog running after restart
|
||||
# validate_unraid — notify validated before use
|
||||
# Silent on success — runs every boot, no noise when already correct
|
||||
# rsyslog Drop Rule
|
||||
# Creates FILTER_FILE (/etc/rsyslog.d/ignore-docker-veth.conf).
|
||||
# Uses RainerScript `stop` action — messages matching "veth" or "docker0" are
|
||||
# dropped before reaching any output target, including the log file.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# FILTER_FILE — path for rsyslog drop filter (default /etc/rsyslog.d/ignore-docker-veth.conf)
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# Writing to /etc/rsyslog.d/ requires root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs at array start.
|
||||
#
|
||||
# rsyslog Process Verify
|
||||
# Confirms rsyslog is running after restart.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs every boot — no noise when filter is already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# FILTER_FILE
|
||||
# Path for the rsyslog drop filter config file.
|
||||
# (default: /etc/rsyslog.d/ignore-docker-veth.conf)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_syslog_filter.sh
|
||||
# Check filter file. Write and restart rsyslog only if changed. Silent if correct.
|
||||
#
|
||||
# docker_syslog_filter.sh --dry-run
|
||||
# Show what would be written without writing or restarting rsyslog.
|
||||
#
|
||||
# docker_syslog_filter.sh --status
|
||||
# Show current filter file content and rsyslog process state.
|
||||
#
|
||||
# docker_syslog_filter.sh --log
|
||||
# Verbose output showing the content comparison and any changes applied.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# docker_syslog_filter.sh — normal run (idempotent)
|
||||
# docker_syslog_filter.sh --dry-run — show what would change
|
||||
# docker_syslog_filter.sh --status — show filter file state and rsyslog status
|
||||
# docker_syslog_filter.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,65 +2,87 @@
|
||||
# ==============================================================================================
|
||||
# ================================= inotify Tuning ============================================
|
||||
# ==============================================================================================
|
||||
# Raises Linux inotify limits at array start to prevent exhaustion across the container stack.
|
||||
# Run once at array start via ARRAY_START_SCRIPTS in master.conf.
|
||||
# Settings are lost on reboot — this script reapplies them on every array start.
|
||||
#
|
||||
# ── THREE INOTIFY LIMITS ──────────────────────────────────────────────────────────────────────
|
||||
# max_user_instances — max number of independent inotify file descriptor objects per user
|
||||
# Each container that calls inotify_init() consumes one instance
|
||||
# Default 128 — exhausted quickly with 20+ active containers
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Raises Linux inotify limits at array start to prevent exhaustion across the
|
||||
# container stack. Run once at array start via ARRAY_START_SCRIPTS. Settings
|
||||
# are lost on reboot — this script reapplies them on every array start.
|
||||
#
|
||||
# max_user_watches — SHARED budget across ALL users and containers on the system
|
||||
# Each watched file or directory costs one watch from this pool
|
||||
# Default 8192 — VSCode alone can need 50K-200K for large workspaces
|
||||
# Must run FIRST in ARRAY_START_SCRIPTS before any containers start — containers
|
||||
# inherit inotify limits at launch, not dynamically. Running this after Code-Server
|
||||
# starts requires a docker restart to pick up the new values.
|
||||
#
|
||||
# max_queued_events — max events buffered before kernel starts dropping them
|
||||
# Low value = events silently lost during high-activity periods
|
||||
# Default 16384 — sufficient for most setups
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── WHY VSCODE THROWS "UNABLE TO WATCH FOR FILE CHANGES" ─────────────────────────────────────
|
||||
# VSCode (and Code-Server in Docker) opens one inotify watch per file in the workspace.
|
||||
# A typical project with node_modules can easily have 100K-200K files.
|
||||
# All containers on the host share max_user_watches — the combined usage of:
|
||||
# Sonarr, Radarr, Lidarr, Emby, Nextcloud, Code-Server, AdGuard, all other arrs
|
||||
# easily exceeds 524288 (512K) watches on a busy server.
|
||||
# Raising to 1048576 (1M) gives sufficient headroom — safe on 128GB RAM (~128MB kernel use).
|
||||
# Three inotify Limits
|
||||
# max_user_instances — max independent inotify fd objects per user; each container
|
||||
# calling inotify_init() consumes one. Default 128 — exhausted
|
||||
# quickly with 20+ active containers.
|
||||
#
|
||||
# ── STARTUP ORDER MATTERS ─────────────────────────────────────────────────────────────────────
|
||||
# inotify_tuning.sh must run BEFORE containers that watch files start.
|
||||
# In ARRAY_START_SCRIPTS order: inotify_tuning.sh first, then container-starting scripts.
|
||||
# If Code-Server starts before limits are raised it inherits the old (low) limits.
|
||||
# Code-Server restart fixes this: limits are kernel-wide, not process-bound at start.
|
||||
# So if Code-Server is already running: docker restart Code-Server after this script runs.
|
||||
# max_user_watches — SHARED budget across ALL users and containers on the system.
|
||||
# Each watched file or directory costs one watch. Default 8192 —
|
||||
# VSCode alone needs 50K–200K for large workspaces. Combined
|
||||
# usage of Sonarr, Radarr, Lidarr, Emby, Nextcloud, Code-Server
|
||||
# easily exceeds 512K on a busy server.
|
||||
#
|
||||
# ── CONSUMERS ON THIS STACK ───────────────────────────────────────────────────────────────────
|
||||
# Emby — watches all media library paths (1 watch per folder)
|
||||
# Sonarr — watches TV_Shows folder tree
|
||||
# Radarr — watches Movies folder tree
|
||||
# Lidarr — watches Music folder tree
|
||||
# Nextcloud — watches data directory for changes
|
||||
# Code-Server — watches entire workspace (can be 50K-200K with node_modules)
|
||||
# AdGuard Home — watches config directory
|
||||
# + all other containers using inotify internally
|
||||
# max_queued_events — max events buffered before the kernel drops them. Low value =
|
||||
# events silently lost during high-activity bursts. Default 16384.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents duplicate runs at array start
|
||||
# Root check — sysctl writes require root
|
||||
# validate_unraid — notify validated before use
|
||||
# Silent on success — runs every boot, no noise when already correct
|
||||
# Only warns on changes or failures
|
||||
# Why 1M Watches
|
||||
# Raising max_user_watches to 1048576 (1M) gives sufficient headroom for all
|
||||
# containers combined — safe on 128GB RAM (~128MB kernel use for the pool).
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# INOTIFY_MAX_INSTANCES — default 1024
|
||||
# INOTIFY_MAX_WATCHES — default 1048576 (1M)
|
||||
# INOTIFY_MAX_QUEUED_EVENTS — default 32768
|
||||
# Idempotent Per-Setting
|
||||
# Each sysctl value is read before writing. Only changed if different from target —
|
||||
# no-op on boots where limits are already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# sysctl writes require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate runs at array start.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs every boot — no noise when already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# INOTIFY_MAX_INSTANCES
|
||||
# Max inotify fd objects per user. (default: 1024)
|
||||
#
|
||||
# INOTIFY_MAX_WATCHES
|
||||
# Max watched files/dirs shared across all users and containers. (default: 1048576)
|
||||
#
|
||||
# INOTIFY_MAX_QUEUED_EVENTS
|
||||
# Max events buffered before kernel drops them. (default: 32768)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# inotify_tuning.sh
|
||||
# Apply inotify limits. No-op per setting if already at target.
|
||||
#
|
||||
# inotify_tuning.sh --dry-run
|
||||
# Show current vs target for each limit. No sysctl writes.
|
||||
#
|
||||
# inotify_tuning.sh --status
|
||||
# Show current vs target, active instance count, and top consumers by PID.
|
||||
#
|
||||
# inotify_tuning.sh --log
|
||||
# Verbose output — show each sysctl check and result.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# inotify_tuning.sh — normal run (apply settings)
|
||||
# inotify_tuning.sh --dry-run — show what would change
|
||||
# inotify_tuning.sh --status — show current vs target values and top consumers
|
||||
# inotify_tuning.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,44 +2,74 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Mover Stop =================================================
|
||||
# ==============================================================================================
|
||||
# Safely stops the unRAID mover process with a warning before halting.
|
||||
# Warns all logged-in users via wall message, waits the configured timeout, then stops.
|
||||
#
|
||||
# ── WHEN TO USE ───────────────────────────────────────────────────────────────────────────────
|
||||
# - Before a planned reboot when mover is running mid-cycle
|
||||
# - Before disk replacement or array operations that need mover stopped
|
||||
# - Before rsync — mover and rsync simultaneously moving the same files causes corruption
|
||||
# - Called automatically by maintenance scripts that need the mover stopped first
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Safely stops the unRAID mover with a wall warning, configurable timeout, and
|
||||
# SIGTERM → SIGKILL sequence. Use before planned reboots, disk operations, or
|
||||
# any operation where mover and rsync running simultaneously could corrupt files.
|
||||
# Exits cleanly if mover is not running.
|
||||
#
|
||||
# ── STOP SEQUENCE ─────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Stop Sequence
|
||||
# 1. Check if mover is running — exit cleanly if not
|
||||
# 2. Broadcast wall warning to all logged-in users
|
||||
# 3. Wait MOVER_STOP_TIMEOUT seconds (default 30) — gives active sessions a chance to note it
|
||||
# 4. Send SIGTERM — mover can complete its current file operation before exiting
|
||||
# 5. Wait 5 seconds for graceful exit
|
||||
# 6. Verify stopped — if still running send SIGKILL (force)
|
||||
# 2. Wall message to all logged-in terminal users
|
||||
# 3. Wait MOVER_STOP_TIMEOUT seconds
|
||||
# 4. SIGTERM — allows mover to finish its current file before stopping
|
||||
# (no partial files — the mover completes what it is working on)
|
||||
# 5. Wait 5 seconds → verify stopped
|
||||
# 6. SIGKILL if still running — forced stop, partial files possible
|
||||
# 7. Final verify — error if still running after SIGKILL
|
||||
#
|
||||
# ── SIGTERM vs SIGKILL ────────────────────────────────────────────────────────────────────────
|
||||
# SIGTERM first — allows mover to finish the file it is currently moving (no partial files).
|
||||
# SIGKILL only as fallback — forces immediate stop (may leave partial files on cache or array).
|
||||
# SIGTERM first because the mover has an opportunity to finish the file it is
|
||||
# currently moving, leaving no partial copies on cache or array. SIGKILL is only
|
||||
# used as a last resort and may leave a file split across cache and array.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent stop attempts racing each other
|
||||
# Root check — pkill on emhttp processes requires root
|
||||
# validate_unraid — notify validated before use
|
||||
# SIGTERM → verify → SIGKILL sequence — graceful then forced
|
||||
# Final verify — confirms mover actually stopped
|
||||
# Silent on clean — mover not running = log() only, no output ✅
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# MOVER_STOP_TIMEOUT — seconds to warn users before stopping (default 30)
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts racing each other.
|
||||
#
|
||||
# Root Required
|
||||
# pkill on emhttp processes requires root.
|
||||
#
|
||||
# Final Verify
|
||||
# Confirms mover is actually stopped after the kill sequence — errors if it
|
||||
# is still running after SIGKILL.
|
||||
#
|
||||
# Silent When Clean
|
||||
# Mover not running = log() only, no visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# MOVER_STOP_TIMEOUT
|
||||
# Seconds between wall warning and SIGTERM. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# mover_stop.sh
|
||||
# Check if mover is running. If so, warn users and stop it.
|
||||
#
|
||||
# mover_stop.sh --dry-run
|
||||
# Show mover state and what would happen. No signals sent.
|
||||
#
|
||||
# mover_stop.sh --status
|
||||
# Show mover state (running, PID, start time). Then exit.
|
||||
#
|
||||
# mover_stop.sh --log
|
||||
# Verbose output showing each step of the stop sequence.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# mover_stop.sh — stop mover with configured timeout
|
||||
# mover_stop.sh --dry-run — show what would happen
|
||||
# mover_stop.sh --status — show mover state
|
||||
# mover_stop.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,53 +2,91 @@
|
||||
# ==============================================================================================
|
||||
# ============================= PHP-FPM Max Children ===========================================
|
||||
# ==============================================================================================
|
||||
# Persistently sets PHP-FPM pm.max_children on unRAID to prevent WebGUI slowdowns.
|
||||
# Run once at array start via ARRAY_START_SCRIPTS in master.conf.
|
||||
# Idempotent — completely silent when value is already correct.
|
||||
#
|
||||
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
|
||||
# unRAID's WebGUI runs through PHP-FPM. The default pm.max_children is very low (4-8).
|
||||
# Under load — multiple users, Docker operations, heavy dashboard usage — all PHP workers
|
||||
# saturate and new requests queue. The WebGUI becomes slow or unresponsive.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Raises PHP-FPM pm.max_children to prevent WebGUI slowdowns under load. Run
|
||||
# once at array start via ARRAY_START_SCRIPTS. Idempotent — silent when the
|
||||
# value is already correct, no restart on clean boot.
|
||||
#
|
||||
# pm.max_children controls how many PHP worker processes can run simultaneously.
|
||||
# Raising it allows the WebGUI to handle more concurrent requests without queuing.
|
||||
# Too high: wastes RAM. Too low: WebGUI slowdowns.
|
||||
# PHP_MAX_CHILDREN=250 is appropriate for 128GB — ~2MB per worker = ~500MB total.
|
||||
# unRAID's WebGUI runs through PHP-FPM. The default pm.max_children is very
|
||||
# low (4–8). Under load — multiple users, Docker operations, heavy dashboard
|
||||
# usage — all PHP workers saturate and new requests queue. The WebGUI becomes
|
||||
# slow or unresponsive.
|
||||
#
|
||||
# ── WHY IDEMPOTENT ────────────────────────────────────────────────────────────────────────────
|
||||
# This runs at every array start. If the value is already correct there is nothing to do —
|
||||
# no config write, no PHP-FPM restart. Restarting PHP-FPM unnecessarily disrupts active
|
||||
# WebGUI sessions and is annoying on every boot.
|
||||
# PHP_MAX_CHILDREN=250 is appropriate for 128GB RAM: ~2MB per worker = ~500MB
|
||||
# total. Too high wastes RAM; too low causes slowdowns.
|
||||
#
|
||||
# ── APPLY SEQUENCE ────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Idempotent Content Check
|
||||
# Reads the current pm.max_children value before writing. If already at
|
||||
# target → silent exit, no PHP-FPM restart. Restarting PHP-FPM unnecessarily
|
||||
# disrupts active WebGUI sessions on every boot.
|
||||
#
|
||||
# Pattern Match Before Write
|
||||
# Verifies the sed pattern finds pm.max_children in the config before
|
||||
# applying any change. Prevents silent failures where sed succeeds but
|
||||
# writes nothing because the key was missing or commented out.
|
||||
#
|
||||
# Apply Sequence
|
||||
# 1. Read current pm.max_children from PHP_CONF
|
||||
# 2. If already at target → exit silently (idempotent)
|
||||
# 2. If already at target → exit silently
|
||||
# 3. Verify sed pattern matches before writing
|
||||
# 4. Apply sed replacement
|
||||
# 5. Restart PHP-FPM via rc.php-fpm
|
||||
# 6. Verify PHP-FPM process running after restart
|
||||
# 7. Verify config file reflects target value
|
||||
# 7. Read back config to confirm value applied
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — writing to system config requires root
|
||||
# acquire_lock — prevents concurrent runs at array start
|
||||
# Idempotent check — only restarts PHP-FPM when value actually changes
|
||||
# Pattern match check — verifies sed found pm.max_children before writing
|
||||
# Process verify — confirms PHP-FPM running after restart
|
||||
# Config verify — reads back config to confirm value applied
|
||||
# validate_unraid — notify validated before use
|
||||
# Silent on correct — runs every boot, no noise when already set ✅
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# PHP_MAX_CHILDREN — target pm.max_children value (default 250)
|
||||
# PHP_CONF — path to PHP-FPM www.conf (default /etc/php83/php-fpm.d/www.conf)
|
||||
# Root Required
|
||||
# Writing to /etc/php83/ requires root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs at array start.
|
||||
#
|
||||
# Process Verify
|
||||
# Confirms PHP-FPM running after restart — errors if it failed to start.
|
||||
#
|
||||
# Config Verify
|
||||
# Reads back config after restart to confirm the value was actually applied.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs every boot — no noise when already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# PHP_MAX_CHILDREN
|
||||
# Target pm.max_children value. (default: 250)
|
||||
#
|
||||
# PHP_CONF
|
||||
# Path to PHP-FPM www.conf. (default: /etc/php83/php-fpm.d/www.conf)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# php_fpm_max_children.sh
|
||||
# Read current value. Update and restart PHP-FPM only if changed. Silent if correct.
|
||||
#
|
||||
# php_fpm_max_children.sh --dry-run
|
||||
# Show current vs target value. No config write or restart.
|
||||
#
|
||||
# php_fpm_max_children.sh --status
|
||||
# Show current pm.max_children, target, and PHP-FPM process state.
|
||||
#
|
||||
# php_fpm_max_children.sh --log
|
||||
# Verbose output showing idempotent check, config write, and restart result.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# php_fpm_max_children.sh — normal run (idempotent)
|
||||
# php_fpm_max_children.sh --dry-run — show what would change
|
||||
# php_fpm_max_children.sh --status — show current vs target and process state
|
||||
# php_fpm_max_children.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,65 +2,105 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Resource Manager ===========================================
|
||||
# ==============================================================================================
|
||||
# Pressure reduction layer — detects rising system load and reduces it intelligently.
|
||||
# Called by watchdog_orchestrator.sh every minute — single-pass, not a continuous loop.
|
||||
#
|
||||
# ── RESPONSIBILITY ────────────────────────────────────────────────────────────────────────────
|
||||
# Reduce system pressure before things break. NOT fixing broken containers (docker_watchdog)
|
||||
# and NOT rebooting (system_watchdog). The middle layer that keeps the system comfortable.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pressure reduction layer — detects rising system load and reduces it before
|
||||
# things break. Called by watchdog_orchestrator.sh every minute as a single-
|
||||
# pass run. The middle layer between docker_watchdog.sh (fixes broken
|
||||
# containers) and system_watchdog.sh (reboots). Does neither of those things.
|
||||
#
|
||||
# "Pressure is rising — reduce load intelligently."
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── THREE-LEVEL PRESSURE RESPONSE ─────────────────────────────────────────────────────────────
|
||||
# Three-Level Pressure Response
|
||||
#
|
||||
# Level 1 — SOFT (RAM < RW_RAM_SOFT_GB OR load > RW_LOAD_SOFT_MULTIPLIER × cores):
|
||||
# Throttle SABnzbd download speed to RW_SABNZBD_SPEED_SOFT
|
||||
# Throttle qBittorrent download to RW_QBIT_DL_SOFT KB/s
|
||||
# Throttle SABnzbd download speed to RW_SABNZBD_SPEED_SOFT.
|
||||
# Throttle qBittorrent download to RW_QBIT_DL_SOFT KB/s.
|
||||
#
|
||||
# Level 2 — MEDIUM (RAM < RW_RAM_MEDIUM_GB OR load > RW_LOAD_MEDIUM_MULTIPLIER × cores):
|
||||
# Further throttle SABnzbd + qBittorrent to medium limits
|
||||
# docker pause RW_PAUSE_CONTAINERS — suspend without losing state, instant reversible
|
||||
# Further throttle SABnzbd + qBittorrent to medium limits.
|
||||
# docker pause RW_PAUSE_CONTAINERS — suspend without losing state, instantly reversible.
|
||||
#
|
||||
# Level 3 — HARD (RAM < RW_RAM_HARD_GB):
|
||||
# docker stop RW_STOP_CONTAINERS — optional/heavy services (games, LocalAI, etc.)
|
||||
# Write mem_shutdown_active=true — signals docker_watchdog to defer container restarts
|
||||
# docker stop RW_STOP_CONTAINERS — optional/heavy services (games, LocalAI, etc.).
|
||||
# Write mem_shutdown_active=true → signals docker_watchdog to defer container restarts.
|
||||
#
|
||||
# ── RECOVERY ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Pressure must stay below current action threshold for RW_RECOVER_CYCLES consecutive runs
|
||||
# before restoring. De-escalates one level at a time to avoid re-triggering immediately.
|
||||
# Level 3 de-escalation additionally requires RAM >= RW_RAM_RECOVER_GB before un-stopping.
|
||||
# Recovery
|
||||
# Pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive
|
||||
# runs before restoring. De-escalates one level at a time — prevents re-triggering
|
||||
# immediately after recovery. Level 3 additionally requires RAM >= RW_RAM_RECOVER_GB
|
||||
# before containers are un-stopped.
|
||||
#
|
||||
# ── COORDINATION WITH DOCKER WATCHDOG ─────────────────────────────────────────────────────────
|
||||
# At level 3: writes mem_shutdown_active=true to RW_STATE_FILE.
|
||||
# Coordination with docker_watchdog.sh
|
||||
# At level 3, writes mem_shutdown_active=true to RW_STATE_FILE.
|
||||
# docker_watchdog.sh reads this and defers all container restart logic.
|
||||
# Cleared when pressure fully resolves and containers are restarted.
|
||||
# This prevents docker_watchdog from restarting containers that RM just stopped to free RAM.
|
||||
# Without this, docker_watchdog would immediately restart containers that were
|
||||
# just stopped to free RAM — defeating the purpose of level 3.
|
||||
# Cleared when pressure resolves and containers are restarted.
|
||||
#
|
||||
# ── NOT RESPONSIBLE FOR ───────────────────────────────────────────────────────────────────────
|
||||
# Restarting broken containers — docker_watchdog.sh
|
||||
# Rebooting the system — system_watchdog.sh
|
||||
# Reacting to single data points — RW_RECOVER_CYCLES prevents flip-flopping
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# Root Required
|
||||
# docker pause/stop require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs from racing on state file writes.
|
||||
#
|
||||
# RW_CRITICAL_CONTAINERS
|
||||
# Containers listed here are never paused or stopped regardless of pressure level.
|
||||
#
|
||||
# RW_ENABLED Flag
|
||||
# Set RW_ENABLED=false to disable the entire script without removing it from
|
||||
# the orchestrator schedule.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
# RW_ENABLED, RW_STATE_FILE
|
||||
# RW_RAM_SOFT_GB, RW_RAM_MEDIUM_GB, RW_RAM_HARD_GB, RW_RAM_RECOVER_GB
|
||||
# RW_LOAD_SOFT_MULTIPLIER, RW_LOAD_MEDIUM_MULTIPLIER
|
||||
# RW_RECOVER_CYCLES
|
||||
# RW_SABNZBD_ENABLED, RW_SABNZBD_SPEED_SOFT, RW_SABNZBD_SPEED_MEDIUM
|
||||
# RW_QBIT_ENABLED, RW_QBIT_DL_SOFT, RW_QBIT_DL_MEDIUM
|
||||
# RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure
|
||||
# RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure (aliased by detect_hosts)
|
||||
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure (aliased by detect_hosts)
|
||||
# master_host*.conf (aliased by detect_hosts())
|
||||
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure
|
||||
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure
|
||||
# HOST*_SABNZBD_URL, HOST*_SABNZBD_API_KEY
|
||||
# HOST*_QBIT_URL, HOST*_QBIT_USERNAME, HOST*_QBIT_PASSWORD
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# resource_watchdog.sh — normal run (via watchdog_orchestrator.sh)
|
||||
# resource_watchdog.sh --dry-run — show what would happen without acting
|
||||
# resource_watchdog.sh --status — current pressure level and active actions
|
||||
# resource_watchdog.sh --log — verbose per-check output
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# RW_STATE_FILE
|
||||
# Pressure level, recovery cycle count, stopped container list, and the
|
||||
# mem_shutdown_active coordination flag read by docker_watchdog.sh.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# resource_watchdog.sh
|
||||
# Single-pass pressure check. Apply actions if threshold crossed. Silent if below.
|
||||
#
|
||||
# resource_watchdog.sh --dry-run
|
||||
# Show current pressure level and what would be throttled/paused/stopped. No changes.
|
||||
#
|
||||
# resource_watchdog.sh --status
|
||||
# Show current pressure level, active actions, recovery cycle count, stopped containers.
|
||||
#
|
||||
# resource_watchdog.sh --log
|
||||
# Verbose per-check output — show RAM, load, each threshold comparison, each action.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,49 +2,87 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Rsync Stop =================================================
|
||||
# ==============================================================================================
|
||||
# Stops rsync intelligently on both local and remote servers.
|
||||
# Auto-detects orchestrators and chooses the safest stop strategy automatically.
|
||||
#
|
||||
# ── TWO MODES ─────────────────────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Stops rsync intelligently on both local and remote servers. Auto-detects
|
||||
# running orchestrators and chooses the safest stop strategy. If an
|
||||
# orchestrator is running, kills only the rsync subprocess so the orchestrator
|
||||
# exits cleanly after finishing the current share. Use --full-stop to kill
|
||||
# everything immediately.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Two Stop Modes
|
||||
# Default (smart):
|
||||
# Detects if an orchestrator (daily/weekly/critical sync) is running
|
||||
# If orchestrator found → kills rsync subprocess only
|
||||
# Orchestrator sees rsync died → moves to next share or exits cleanly
|
||||
# If no orchestrator → kills rsync directly (standalone rsync.sh run)
|
||||
# Cleans stale lock files after kill
|
||||
# Recovers containers left stopped by interrupted rsync (local only)
|
||||
# Detects if an orchestrator (daily/weekly/critical sync) is running.
|
||||
# If orchestrator found → kills rsync subprocess only. Orchestrator sees
|
||||
# rsync exit → moves to next share or exits cleanly on its own.
|
||||
# If no orchestrator → kills rsync directly (standalone rsync.sh run).
|
||||
# Cleans stale lock files after kill.
|
||||
# Recovers containers left stopped by interrupted rsync (local only).
|
||||
#
|
||||
# --full-stop (nuclear):
|
||||
# Kills orchestrator first → then kills rsync
|
||||
# Orchestrator will NOT continue to next share
|
||||
# Use when: you need everything dead immediately
|
||||
# Kills orchestrator first → then kills rsync.
|
||||
# Orchestrator will NOT continue to next share.
|
||||
# Use when everything needs to stop immediately.
|
||||
#
|
||||
# ── REMOTE HANDLING ───────────────────────────────────────────────────────────────────────────
|
||||
# Both local and remote handled in one run via SSH.
|
||||
# Remote containers left as-is — docker_watchdog.sh handles remote container recovery.
|
||||
# If remote unreachable → skips remote cleanly, logs warning.
|
||||
# Orchestrator Detection
|
||||
# detect_rsync_parent() scans all lock files to find which running process
|
||||
# has rsync as a descendant. No hardcoded list — works for any orchestrator.
|
||||
# Returns "script_name:parent_pid" if found, empty if standalone.
|
||||
#
|
||||
# ── ORCHESTRATOR DETECTION ────────────────────────────────────────────────────────────────────
|
||||
# detect_rsync_parent() scans all lock files to find which running process
|
||||
# has rsync as a descendant. No hardcoded list — works for any orchestrator.
|
||||
# Returns: "script_name:parent_pid" if found, empty if rsync running standalone.
|
||||
# Remote Handling
|
||||
# Both local and remote handled in one run via SSH.
|
||||
# Remote containers left as-is — docker_watchdog.sh handles remote recovery.
|
||||
# If remote unreachable → skips remote cleanly, logs warning.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — pkill and docker require root
|
||||
# acquire_lock — prevents concurrent stop attempts racing
|
||||
# DOCKER_TIMEOUT — all docker calls protected against hung daemon
|
||||
# SSH_TIMEOUT — all remote SSH calls timeout-protected
|
||||
# SIGTERM → SIGKILL — graceful then forced for orchestrators
|
||||
# Container recovery — restarts local containers left stopped by killed rsync
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# pkill and docker require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts racing each other.
|
||||
#
|
||||
# Timeout Protection
|
||||
# DOCKER_TIMEOUT (15s) on all docker calls — hung daemon doesn't block.
|
||||
# SSH_TIMEOUT (15s) on all remote SSH calls.
|
||||
#
|
||||
# SIGTERM → SIGKILL Sequence
|
||||
# Orchestrators receive SIGTERM first, SIGKILL only if still running after 2s.
|
||||
#
|
||||
# Container Recovery
|
||||
# Restarts local containers left stopped by the killed rsync session.
|
||||
# Remote containers deferred to docker_watchdog.sh.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# rsync_stop.sh
|
||||
# Auto-detect orchestrator. Kill rsync-only or full-stop accordingly.
|
||||
#
|
||||
# rsync_stop.sh --full-stop
|
||||
# Kill orchestrator first, then kill rsync. Nothing continues after this.
|
||||
#
|
||||
# rsync_stop.sh --rsync-only
|
||||
# Skip container recovery. Used when called by other scripts that handle
|
||||
# recovery themselves.
|
||||
#
|
||||
# rsync_stop.sh --dry-run
|
||||
# Show what would be killed without killing anything.
|
||||
#
|
||||
# rsync_stop.sh --status
|
||||
# Show local and remote rsync PIDs, running orchestrators, and lock files.
|
||||
#
|
||||
# rsync_stop.sh --full-stop --dry-run
|
||||
# Preview full-stop sequence without making any changes.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# rsync_stop.sh — smart stop (auto-detect)
|
||||
# rsync_stop.sh --full-stop — kill orchestrator + rsync
|
||||
# rsync_stop.sh --rsync-only — skip container recovery (called by other scripts)
|
||||
# rsync_stop.sh --dry-run — preview without changes
|
||||
# rsync_stop.sh --status — show what's currently running
|
||||
# rsync_stop.sh --full-stop --dry-run — preview full stop
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,55 +2,88 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Server Reboot ==============================================
|
||||
# ==============================================================================================
|
||||
# Gracefully reboots the unRAID server with full pre-flight checks and clean shutdown sequence.
|
||||
# Warns all users, checks for active processes, stops services, syncs disks, then reboots.
|
||||
#
|
||||
# ── SHUTDOWN SEQUENCE ─────────────────────────────────────────────────────────────────────────
|
||||
# 1. Pre-flight warnings — rsync, mover, active Emby sessions (warn not block)
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Gracefully reboots the unRAID server with pre-flight checks, user warnings,
|
||||
# clean service shutdown, and disk sync. Use instead of raw /sbin/reboot —
|
||||
# gives users warning time and ensures services stop cleanly before the kernel
|
||||
# drops.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Shutdown Sequence
|
||||
# 1. Pre-flight warnings — rsync, mover, active Emby sessions (warn, not block)
|
||||
# 2. Wall message to all logged-in terminal users
|
||||
# 3. unRAID notification to dashboard
|
||||
# 4. Wait REBOOT_SLEEP seconds (default 30) — gives users time to save work
|
||||
# 5. Gracefully shutdown VMs (virsh shutdown each, then wait)
|
||||
# 3. unRAID dashboard notification
|
||||
# 4. Wait REBOOT_SLEEP seconds — users time to save work
|
||||
# 5. Graceful VM shutdown via virsh — ACPI signal, then wait REBOOT_VM_WAIT
|
||||
# 6. Stop libvirt (VM Manager)
|
||||
# 7. Stop Docker service
|
||||
# 8. Sync filesystem buffers to disk
|
||||
# 9. Reboot
|
||||
# 8. sync — filesystem buffers flushed to disk
|
||||
# 9. /sbin/reboot
|
||||
#
|
||||
# ── PRE-FLIGHT WARNINGS ───────────────────────────────────────────────────────────────────────
|
||||
# The following are warnings only — they do not block the reboot. You called this script,
|
||||
# so you know what you're doing. The warnings give you context before the countdown starts.
|
||||
# - rsync running → partial files possible if mid-transfer
|
||||
# - mover running → files may be left on cache or array mid-move
|
||||
# - Emby sessions → active streams/transcodes will be interrupted
|
||||
# Pre-flight Warnings (informational — do not block)
|
||||
# rsync running → partial files possible if mid-transfer
|
||||
# mover running → files may be left mid-move on cache or array
|
||||
# Emby sessions → active streams/transcodes interrupted
|
||||
# Warnings do not block the reboot — you called this script, you know.
|
||||
#
|
||||
# ── VM GRACEFUL SHUTDOWN ──────────────────────────────────────────────────────────────────────
|
||||
# virsh shutdown sends ACPI power button signal to each VM — same as pressing power button.
|
||||
# VM gets a chance to flush its own buffers and shutdown cleanly.
|
||||
# Waits REBOOT_VM_WAIT seconds (default 30) for VMs to shut down before stopping libvirt.
|
||||
# If VMs don't shut down in time libvirt stops anyway — system reboot takes priority.
|
||||
# VM Graceful Shutdown
|
||||
# virsh shutdown sends the ACPI power button signal — same as pressing the
|
||||
# physical power button. VM gets a chance to flush buffers and shut down.
|
||||
# After REBOOT_VM_WAIT seconds, libvirt stops anyway — reboot takes priority.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID — used in wall message, notification, and summary.
|
||||
# Critical on a two-server setup — wall and notifications show WHICH server is rebooting.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — reboot requires root
|
||||
# acquire_lock — prevents concurrent reboot calls
|
||||
# detect_hosts() — MY_ID in all user-facing messages
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Graceful VM shutdown — VMs get clean ACPI signal before libvirt stops
|
||||
# sync before reboot — filesystem buffers flushed to disk
|
||||
# Root Required
|
||||
# /sbin/reboot requires root.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# REBOOT_SLEEP — seconds to warn users before starting shutdown sequence (default 30)
|
||||
# REBOOT_VM_WAIT — seconds to wait for VMs to shut down gracefully (default 30)
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent reboot calls.
|
||||
#
|
||||
# Host Identity in All Messages
|
||||
# detect_hosts() sets MY_ID — wall and notifications show which server is
|
||||
# rebooting. Critical on a two-server setup.
|
||||
#
|
||||
# sync Before Reboot
|
||||
# filesystem buffers flushed to disk before reboot command.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# REBOOT_SLEEP
|
||||
# Seconds between warning and shutdown sequence start. (default: 30)
|
||||
#
|
||||
# REBOOT_VM_WAIT
|
||||
# Seconds to wait for VMs to shut down gracefully. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# server_reboot.sh
|
||||
# Run pre-flight, warn users, stop services cleanly, then reboot.
|
||||
#
|
||||
# server_reboot.sh --dry-run
|
||||
# Walk through the entire shutdown sequence without stopping anything or rebooting.
|
||||
#
|
||||
# server_reboot.sh --status
|
||||
# Show running processes that would be affected: rsync, mover, VMs, containers.
|
||||
#
|
||||
# server_reboot.sh --reason="maintenance"
|
||||
# Include reason in wall message and notification. Defaults to "manual".
|
||||
#
|
||||
# server_reboot.sh --log
|
||||
# Verbose output — show each step of the shutdown sequence.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# server_reboot.sh — reboot with 30s warning
|
||||
# server_reboot.sh --dry-run — walk through sequence without rebooting
|
||||
# server_reboot.sh --status — show running processes that would be affected
|
||||
# server_reboot.sh --reason="maintenance" — log reason for reboot
|
||||
# server_reboot.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,71 +2,107 @@
|
||||
# ==============================================================================================
|
||||
# ================================= System Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Last line of defense — reboots the system cleanly if it is about to become unstable.
|
||||
# Runs continuously as a background process — started by array_started.sh at array start.
|
||||
# Works alongside docker_watchdog.sh which handles container-level healing first.
|
||||
#
|
||||
# ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Last line of defense — reboots the system cleanly if it is about to become
|
||||
# unstable. Runs continuously as a background process started by
|
||||
# array_started.sh at array start. Works alongside docker_watchdog.sh which
|
||||
# handles container-level healing first. Only escalates to reboot when
|
||||
# docker_watchdog.sh cannot resolve the condition.
|
||||
#
|
||||
# TIER 1 — CRITICAL (bypass ALL strikes, reboot immediately)
|
||||
# Docker daemon unresponsive — nothing can be healed, letting it run makes it worse
|
||||
# rootfs at 99%+ — writes failing, SSH may stop, no recovery options
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three-Tier Response System
|
||||
#
|
||||
# Tier 1 — CRITICAL (bypass all strikes, reboot immediately)
|
||||
# Docker daemon unresponsive — nothing can be healed; running it longer makes it worse
|
||||
# rootfs at 99%+ — writes failing; SSH may stop; no recovery options
|
||||
# Kernel oops/BUG in dmesg — kernel running with corrupted state
|
||||
# File descriptor exhaustion — new connections and processes failing silently
|
||||
# /boot read-only unexpectedly — state files and config writes silently failing
|
||||
#
|
||||
# TIER 2 — URGENT (bypass strikes when OOM confirms active crisis)
|
||||
# RAM < MEM_GB AND OOM kills >= OOM_LIMIT in this cycle
|
||||
# Rationale: OOM kills at this rate means system is dying faster than watchdogs heal
|
||||
# Without OOM confirmation → standard strike system applies
|
||||
# Tier 2 — URGENT (bypass strikes when OOM confirms active crisis)
|
||||
# RAM < MEM_GB AND OOM kills >= OOM_LIMIT in this cycle.
|
||||
# OOM kills at this rate means the system is dying faster than watchdogs can heal.
|
||||
# Without OOM confirmation → standard strike system applies.
|
||||
#
|
||||
# TIER 3 — STANDARD (N consecutive failures → reboot)
|
||||
# RAM tiers, load, CPU temp, zombies, /var/log, /tmp, containers, NIC, mdstat
|
||||
# Tier 3 — STANDARD (N consecutive failures → reboot)
|
||||
# RAM tiers, load, CPU temp, zombies, /var/log, /tmp, containers, NIC, mdstat.
|
||||
#
|
||||
# ── RAM TIERS ─────────────────────────────────────────────────────────────────────────────────
|
||||
# RAM Tiers
|
||||
# MEM_WARN_GB (10GB) — warn + notify only
|
||||
# MEM_SHUTDOWN_GB (6GB) — stop non-essential containers, wait for recovery
|
||||
# MEM_GB (4GB) — strike system → reboot (bypass if OOM confirms)
|
||||
# MEM_RECOVER_GB (30GB) — RAM must reach this before containers restart
|
||||
# MEM_GB (4GB) — strike system → reboot (bypass with OOM confirmation)
|
||||
# MEM_RECOVER_GB (30GB) — RAM must reach this before stopped containers restart
|
||||
#
|
||||
# ── CONTAINER SHUTDOWN LOGIC ──────────────────────────────────────────────────────────────────
|
||||
# At MEM_SHUTDOWN_GB: stop all containers NOT in SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED
|
||||
# Excluded: NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr
|
||||
# Stopped containers tracked in shutdown list — won't restart until RAM recovers
|
||||
# Strike system prevents flip-flopping — shutdown only happens once per degradation event
|
||||
# Container Shutdown Logic (at MEM_SHUTDOWN_GB)
|
||||
# Stops all containers not in SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED.
|
||||
# Stopped containers tracked in shutdown list — won't restart until RAM recovers.
|
||||
# Strike system prevents flip-flopping — shutdown only once per degradation event.
|
||||
#
|
||||
# ── OOM TRACKING ──────────────────────────────────────────────────────────────────────────────
|
||||
# /proc/vmstat oom_kill counter — read each cycle, delta = kills this cycle
|
||||
# Included in reboot message with process names from dmesg (diagnostic context)
|
||||
# Bypass trigger: RAM critical AND kills this cycle >= SYS_WATCHDOG_OOM_LIMIT
|
||||
# Abort Conditions (prevent reboot during sensitive operations)
|
||||
# ZFS pool unhealthy, parity running, mover running — each toggleable.
|
||||
# CRITICAL tier bypasses all abort conditions — imminent crash overrides data safety.
|
||||
#
|
||||
# ── NEW CHECKS THIS VERSION ───────────────────────────────────────────────────────────────────
|
||||
# OOM rate tracking — delta from /proc/vmstat each cycle
|
||||
# /boot read-only — write test on /boot each cycle
|
||||
# Kernel oops detection — dmesg BUG/Oops count delta each cycle
|
||||
# File descriptor exhaustion — /proc/sys/fs/file-nr utilisation
|
||||
# /tmp usage — tmpfs fill detection with auto-clear attempt
|
||||
# Array disk errors — mdstat error delta each cycle
|
||||
# Runaway process — single process >N% CPU sustained (disabled by default)
|
||||
# NIC state check — primary interface operstate
|
||||
# sshd check — restart attempt before escalating
|
||||
# Checks Run Every Cycle
|
||||
# rootfs usage, /var/log, /tmp, free RAM, ZFS ARC, CPU temp, load avg,
|
||||
# zombie processes, Docker daemon, OOM rate, /boot read-only, kernel oops,
|
||||
# file descriptor exhaustion, array disk errors, NIC state, required containers.
|
||||
#
|
||||
# ── EXISTING CHECKS ───────────────────────────────────────────────────────────────────────────
|
||||
# rootfs usage, /var/log, free RAM, ZFS ARC, CPU temp, load avg,
|
||||
# zombie processes, Docker daemon, required containers from skip list
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── ABORT CONDITIONS ──────────────────────────────────────────────────────────────────────────
|
||||
# ZFS pool unhealthy, parity running, mover running — toggleable
|
||||
# CRITICAL tier bypasses abort conditions — imminent crash overrides data safety
|
||||
# Root Required
|
||||
# Reboot and container stop require root.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# Full config under System Watchdog section — see master.conf for all vars
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents a second watchdog instance from starting.
|
||||
#
|
||||
# State File Verification
|
||||
# All state files verified writable at startup — errors if any cannot be created.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf — System Watchdog section
|
||||
# Full variable listing in master.conf. Key variables:
|
||||
#
|
||||
# SYS_WATCHDOG_REBOOT_WINDOW_HRS — reboot rate limit window (default: 2)
|
||||
# SYS_WATCHDOG_MAX_REBOOTS — max reboots in window before giving up (default: 3)
|
||||
# SYS_WATCHDOG_STRIKES — consecutive failures before reboot (default: 3)
|
||||
# SYS_WATCHDOG_OOM_LIMIT — OOM kills/cycle to trigger URGENT bypass (default: 3)
|
||||
# SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED — containers exempt from memory shutdown
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# SYS_WATCHDOG_STATE_FILE — strike counters and cycle state
|
||||
# SYS_WATCHDOG_REBOOT_LOG — reboot history for rate limiting
|
||||
# SYS_WATCHDOG_FAILED_FILE — containers confirmed down for skip list integration
|
||||
# SYS_WATCHDOG_OOM_FILE — OOM kill counter from previous cycle
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# system_watchdog.sh
|
||||
# Start continuous monitoring loop. Runs until stopped or system reboots.
|
||||
#
|
||||
# system_watchdog.sh --dry-run
|
||||
# Run detection logic without rebooting or stopping containers.
|
||||
#
|
||||
# system_watchdog.sh --status
|
||||
# Show config, thresholds, current system state, and strike counts.
|
||||
#
|
||||
# system_watchdog.sh --log
|
||||
# Verbose per-cycle output — show every check result and threshold comparison.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# system_watchdog.sh — normal start (continuous loop)
|
||||
# system_watchdog.sh --dry-run — trigger detection without rebooting
|
||||
# system_watchdog.sh --status — show config and thresholds
|
||||
# system_watchdog.sh --log — verbose per-cycle output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,45 +2,70 @@
|
||||
# ==============================================================================================
|
||||
# ============================= User Scripts Stop ==============================================
|
||||
# ==============================================================================================
|
||||
# Stops all running User Script processes spawned by the unRAID User Scripts plugin.
|
||||
# Identifies processes by their /tmp/user.scripts path signature.
|
||||
# Shows script names not just PIDs — you know what's being stopped.
|
||||
#
|
||||
# ── WHEN TO USE ───────────────────────────────────────────────────────────────────────────────
|
||||
# - Before a planned reboot when scripts are running mid-cycle
|
||||
# - When a script is stuck and won't respond to the Abort button in the UI
|
||||
# - Called automatically by server_reboot.sh as part of shutdown sequence
|
||||
# - Emergency stop of all background ecosystem scripts
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Stops all running User Script processes spawned by the unRAID User Scripts
|
||||
# plugin. Shows script names not just PIDs so you know what's being stopped.
|
||||
# Called automatically by server_reboot.sh as part of the shutdown sequence,
|
||||
# and useful directly when a script is stuck and won't respond to the UI.
|
||||
#
|
||||
# ── HOW IT IDENTIFIES PROCESSES ───────────────────────────────────────────────────────────────
|
||||
# Scans /proc/*/cmdline for processes whose command line contains "/tmp/user.scripts".
|
||||
# The unRAID User Scripts plugin stages all scripts in /tmp/user.scripts/ before execution.
|
||||
# This is more reliable than process name matching which can vary.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── STOP SEQUENCE PER PROCESS ─────────────────────────────────────────────────────────────────
|
||||
# 1. Send SIGTERM — allows script to trap and clean up gracefully
|
||||
# Process Identification
|
||||
# Scans /proc/*/cmdline for processes whose command line contains
|
||||
# "/tmp/user.scripts". The User Scripts plugin stages all scripts in
|
||||
# /tmp/user.scripts/ before execution — more reliable than process name
|
||||
# matching which can vary.
|
||||
#
|
||||
# Stop Sequence Per Process
|
||||
# 1. Send SIGTERM — allows the script to trap and clean up gracefully
|
||||
# 2. Wait 5 seconds
|
||||
# 3. Check if still running → SIGKILL (force) if SIGTERM ignored
|
||||
# 4. Verify dead after SIGKILL
|
||||
# 3. If still running → SIGKILL (force)
|
||||
# 4. Verify dead after SIGKILL — error if still running
|
||||
#
|
||||
# ── SELF-EXCLUSION ────────────────────────────────────────────────────────────────────────────
|
||||
# If this script itself is run via the User Scripts plugin it would find its own PID.
|
||||
# Self-exclusion prevents this script from killing itself mid-execution.
|
||||
# Self-Exclusion
|
||||
# If this script is run via the User Scripts plugin it would find its own
|
||||
# PID in the scan. Self-exclusion by PID prevents killing its own process
|
||||
# tree mid-execution.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — kill requires root for other users' processes
|
||||
# acquire_lock — prevents concurrent stop attempts
|
||||
# Self-exclusion — never kills its own process tree
|
||||
# SIGTERM → SIGKILL — graceful then forced
|
||||
# Verify after kill — confirms processes are actually dead
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Silent when clean — no processes running = log() only ✅
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# kill requires root for other users' processes.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts.
|
||||
#
|
||||
# SIGTERM → SIGKILL Sequence
|
||||
# Graceful first. Forced only if SIGTERM ignored after 5 seconds.
|
||||
#
|
||||
# Post-Kill Verify
|
||||
# Confirms each process is actually dead. Errors and notifies if unkillable.
|
||||
#
|
||||
# Silent When Clean
|
||||
# No processes running = log() only, no visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# user_scripts_stop.sh
|
||||
# Find and stop all User Script processes. Silent if none running.
|
||||
#
|
||||
# user_scripts_stop.sh --dry-run
|
||||
# Show which processes would be stopped, with names and runtimes. No kills.
|
||||
#
|
||||
# user_scripts_stop.sh --status
|
||||
# Show currently running User Script processes with names and elapsed time.
|
||||
#
|
||||
# user_scripts_stop.sh --log
|
||||
# Verbose output — show each process found, each signal sent, each result.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# user_scripts_stop.sh — stop all user scripts
|
||||
# user_scripts_stop.sh --dry-run — show what would be stopped
|
||||
# user_scripts_stop.sh --status — show currently running user scripts
|
||||
# user_scripts_stop.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,57 +2,92 @@
|
||||
# ==============================================================================================
|
||||
# ================================= WebGUI Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Monitors the unRAID WebGUI and restarts services if unresponsive.
|
||||
# Uses a three-step escalating strategy — lightest fix first, heaviest last.
|
||||
# Run every 5-10 minutes via User Scripts plugin.
|
||||
# Silent when healthy — only produces output when something needs fixing.
|
||||
#
|
||||
# ── ESCALATION PATH ───────────────────────────────────────────────────────────────────────────
|
||||
# Check WebGUI → responding → log() + exit 0 (completely silent ✅)
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Monitors the unRAID WebGUI and restarts services if unresponsive. Uses a
|
||||
# three-step escalating strategy — lightest fix first, heaviest last. Run
|
||||
# every 5–10 minutes via the User Scripts plugin. Silent when healthy.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Escalation Path
|
||||
# WebGUI responding → log() + exit 0 (completely silent ✅)
|
||||
#
|
||||
# Not responding:
|
||||
# Step 1 — Restart nginx
|
||||
# Lightest fix — handles most transient WebGUI failures
|
||||
# nginx crash, worker stuck, connection timeout
|
||||
# Wait WEBGUI_NGINX_WAIT seconds → recheck
|
||||
# Step 1 — nginx restart
|
||||
# Lightest fix — handles most transient WebGUI failures:
|
||||
# nginx crash, worker stuck, connection timeout.
|
||||
# Wait WEBGUI_NGINX_WAIT seconds → recheck.
|
||||
#
|
||||
# Step 2 — Restart php-fpm
|
||||
# WebGUI runs through PHP-FPM — worker exhaustion causes silent failure
|
||||
# php-fpm workers saturated → new requests queue → WebGUI appears frozen
|
||||
# system_tuning_monitor.sh tracks usage — this recovers it
|
||||
# Wait WEBGUI_PHP_WAIT seconds → recheck
|
||||
# Step 2 — php-fpm restart
|
||||
# WebGUI runs through PHP-FPM. Worker exhaustion causes silent
|
||||
# failure — requests queue and the WebGUI appears frozen.
|
||||
# Wait WEBGUI_PHP_WAIT seconds → recheck.
|
||||
#
|
||||
# Step 3 — Restart emhttp
|
||||
# Heaviest fix — emhttp is the unRAID management daemon
|
||||
# Array, Docker, shares stay running — only WebGUI management restarts
|
||||
# Takes longer to recover — WEBGUI_EMHTTP_WAIT gives it time
|
||||
# Wait WEBGUI_EMHTTP_WAIT seconds → recheck
|
||||
# Step 3 — emhttp restart
|
||||
# Heaviest fix. emhttp is the unRAID management daemon.
|
||||
# Array, Docker, and shares stay running — only WebGUI
|
||||
# management restarts. Takes longer — WEBGUI_EMHTTP_WAIT.
|
||||
# Wait WEBGUI_EMHTTP_WAIT seconds → recheck.
|
||||
#
|
||||
# All three failed → notify warning, manual intervention needed → exit 1
|
||||
# All three failed → notify, manual intervention needed → exit 1.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID — used in all notifications and summary.
|
||||
# Critical on two-server setup — which server's WebGUI failed?
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent runs double-restarting services
|
||||
# detect_hosts() — MY_ID in all notifications
|
||||
# Process verify — pgrep check after each service restart
|
||||
# Silent healthy — completely silent on healthy cycle ✅
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Root Required
|
||||
# Service restart commands require root.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# WEBGUI_URL — URL to check (default http://localhost)
|
||||
# WEBGUI_TIMEOUT — curl timeout in seconds (default 5)
|
||||
# WEBGUI_NGINX_WAIT — seconds after nginx restart before rechecking (default 15)
|
||||
# WEBGUI_PHP_WAIT — seconds after php-fpm restart before rechecking (default 10)
|
||||
# WEBGUI_EMHTTP_WAIT — seconds after emhttp restart before rechecking (default 30)
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs double-restarting services.
|
||||
#
|
||||
# Process Verify After Each Restart
|
||||
# pgrep check after each rc.* command — errors if process not running.
|
||||
#
|
||||
# Silent When Healthy
|
||||
# Completely silent on healthy cycles. Only produces output when recovering.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WEBGUI_URL
|
||||
# URL to check for WebGUI response. (default: http://localhost)
|
||||
#
|
||||
# WEBGUI_TIMEOUT
|
||||
# curl timeout in seconds. (default: 5)
|
||||
#
|
||||
# WEBGUI_NGINX_WAIT
|
||||
# Seconds after nginx restart before rechecking. (default: 15)
|
||||
#
|
||||
# WEBGUI_PHP_WAIT
|
||||
# Seconds after php-fpm restart before rechecking. (default: 10)
|
||||
#
|
||||
# WEBGUI_EMHTTP_WAIT
|
||||
# Seconds after emhttp restart before rechecking. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# webgui_restart.sh
|
||||
# Check WebGUI. Escalate through nginx → php-fpm → emhttp if unresponsive.
|
||||
#
|
||||
# webgui_restart.sh --dry-run
|
||||
# Show which services would be restarted. No restarts, no waits.
|
||||
#
|
||||
# webgui_restart.sh --status
|
||||
# Show current WebGUI response state and nginx/php-fpm/emhttp process states.
|
||||
#
|
||||
# webgui_restart.sh --log
|
||||
# Verbose output — show each check, each restart attempt, each wait.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# webgui_restart.sh — check and recover if needed
|
||||
# webgui_restart.sh --dry-run — show what would be restarted
|
||||
# webgui_restart.sh --status — show current WebGUI and service states
|
||||
# webgui_restart.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user