Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename) - Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API for storage health, service management, mover, user scripts, notifications, disk temps, and platform command validation - Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR, auto-source Plugin/$PLATFORM/adapter.sh after common.sh - Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg, disks.ini, and validate_unraid_cmd calls with platform_*() functions across watchdogs, orchestrators, and System_Essentials scripts - Update all documentation: rename refs, update webgui escalation logic, add platform adapter section to Plugin README, update main README with portability vision and corrected self-healing stack description
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,556 @@
|
||||
# ━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━
|
||||
|
||||
Configuration reference, operational procedures, and troubleshooting for
|
||||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||||
adding or reordering scripts at array start.
|
||||
|
||||
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
|
||||
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
|
||||
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||||
- [webgui_watchdog.sh](#webgui_watchdogsh)
|
||||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||||
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
|
||||
- [clear_logs.sh](#clear_logssh)
|
||||
- [mover_stop.sh](#mover_stopsh)
|
||||
- [rsync_stop.sh](#rsync_stopsh)
|
||||
- [user_scripts_stop.sh](#user_scripts_stopsh)
|
||||
- [server_reboot.sh](#server_rebootsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
---
|
||||
|
||||
## Output Tiers
|
||||
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
|
||||
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
|
||||
Per-check detail suppressed.
|
||||
|
||||
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
|
||||
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
|
||||
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
|
||||
final summary are visible. Per-item detail suppressed.
|
||||
|
||||
---
|
||||
|
||||
## ARRAY_START_SCRIPTS Order
|
||||
|
||||
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
|
||||
> scripts.** Getting it wrong causes subtle failures that don't show up
|
||||
> immediately.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
|
||||
# any container starts. Containers inherit
|
||||
# inotify limits at launch, not dynamically.
|
||||
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
|
||||
# created. If a container starts first, its
|
||||
# veth creation is already in syslog.
|
||||
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
|
||||
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
|
||||
...
|
||||
)
|
||||
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
|
||||
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
|
||||
```
|
||||
|
||||
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
|
||||
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
|
||||
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
|
||||
|
||||
Why docker_syslog_filter SECOND: The filter must be in place before any container
|
||||
starts creating veth interfaces. The first container start after array start
|
||||
generates veth messages — these will appear in syslog if the filter isn't active.
|
||||
|
||||
---
|
||||
|
||||
## webgui_watchdog.sh
|
||||
|
||||
### Escalation Logic
|
||||
|
||||
```
|
||||
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
|
||||
|
||||
Not responding:
|
||||
1. platform_restart_service nginx
|
||||
wait WEBGUI_NGINX_WAIT (15s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
2. platform_restart_service php-fpm
|
||||
wait WEBGUI_PHP_WAIT (10s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
3. platform_restart_service emhttp
|
||||
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
All three failed → notify warning, exit 1
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
WEBGUI_URL="http://localhost" # URL to check
|
||||
WEBGUI_TIMEOUT=5 # curl timeout in seconds
|
||||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
|
||||
```
|
||||
|
||||
### WebGUI Frozen — Manual Recovery
|
||||
|
||||
```bash
|
||||
# Check which services are running:
|
||||
webgui_watchdog.sh --status
|
||||
|
||||
# Try manual restart sequence (mirrors what the script calls):
|
||||
# Source the ecosystem first to get platform functions:
|
||||
source /boot/config/plugins/varaverk/load_config.sh
|
||||
platform_restart_service nginx
|
||||
# wait 15s, then:
|
||||
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
|
||||
|
||||
# If nginx did not fix it, php-fpm:
|
||||
platform_restart_service php-fpm
|
||||
|
||||
# If still down, emhttp:
|
||||
platform_restart_service emhttp
|
||||
|
||||
# Raw equivalents (no source needed — paste directly into terminal):
|
||||
# /etc/rc.d/rc.nginx restart
|
||||
# /etc/rc.d/rc.php-fpm restart
|
||||
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
|
||||
|
||||
# If all three failed:
|
||||
server_reboot.sh --status # check for active sessions first
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## inotify_tuning.sh
|
||||
|
||||
### What It Sets
|
||||
|
||||
```bash
|
||||
INOTIFY_MAX_INSTANCES=1024 # max inotify fd objects per user (default: 128)
|
||||
INOTIFY_MAX_WATCHES=1048576 # max watches shared across all users (default: 8192)
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768 # max buffered events (default: 16384)
|
||||
```
|
||||
|
||||
Verify current values:
|
||||
|
||||
```bash
|
||||
inotify_tuning.sh --status
|
||||
# Shows current vs target for each limit, active instance count, top consumers
|
||||
```
|
||||
|
||||
### If Code-Server Shows "Unable to Watch for File Changes"
|
||||
|
||||
```bash
|
||||
# 1. Verify limits are set:
|
||||
sysctl fs.inotify.max_user_watches
|
||||
# Expected: 1048576
|
||||
|
||||
# 2. Check total usage across all containers:
|
||||
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l
|
||||
|
||||
# 3. If limits are set but Code-Server still shows the error:
|
||||
docker restart Code-Server
|
||||
# Running containers inherit limits at launch. Restart picks up the new values.
|
||||
|
||||
# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
|
||||
inotify_tuning.sh --log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## php_fpm_max_children.sh
|
||||
|
||||
### What It Sets
|
||||
|
||||
```bash
|
||||
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
```
|
||||
|
||||
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
|
||||
The default of 4–8 saturates immediately under load on a busy server.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
php_fpm_max_children.sh --status
|
||||
# Shows current value vs target, PHP-FPM worker count
|
||||
|
||||
# Manual verify:
|
||||
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
|
||||
# Expected: pm.max_children = 250
|
||||
```
|
||||
|
||||
### If WebGUI Is Slow Despite the Setting
|
||||
|
||||
```bash
|
||||
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
|
||||
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
|
||||
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
|
||||
# pm.max_children value may still be too low, or there's a PHP worker leak.
|
||||
|
||||
# Check running worker count:
|
||||
pgrep -fc php-fpm
|
||||
# Compare to pm.max_children — if equal, workers are saturated
|
||||
|
||||
# Increase if needed:
|
||||
# master.conf: PHP_MAX_CHILDREN=350
|
||||
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## docker_syslog_filter.sh
|
||||
|
||||
### What It Creates
|
||||
|
||||
```
|
||||
/etc/rsyslog.d/ignore-docker-veth.conf:
|
||||
if ($msg contains "veth" or $msg contains "docker0") then {
|
||||
stop
|
||||
}
|
||||
```
|
||||
|
||||
This drops any syslog message containing "veth" or "docker0" before it reaches
|
||||
any output target, including the log file. The drop rule is applied at rsyslog
|
||||
level — not at the log viewer level.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
docker_syslog_filter.sh --status
|
||||
# Shows filter file content and rsyslog process state
|
||||
|
||||
# Manual verify:
|
||||
cat /etc/rsyslog.d/ignore-docker-veth.conf
|
||||
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"
|
||||
|
||||
# Test the filter is active (should produce no syslog output):
|
||||
logger "test veth message"
|
||||
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
|
||||
```
|
||||
|
||||
### If Syslog Still Has Veth Noise
|
||||
|
||||
```bash
|
||||
# 1. Verify filter file exists with correct content:
|
||||
docker_syslog_filter.sh --status
|
||||
|
||||
# 2. If content differs — re-apply:
|
||||
docker_syslog_filter.sh --log
|
||||
|
||||
# 3. Verify rsyslog is using the conf.d directory:
|
||||
grep -r "IncludeConfig" /etc/rsyslog.conf
|
||||
# Expected: IncludeConfig /etc/rsyslog.d/*.conf (or similar)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## clear_logs.sh
|
||||
|
||||
### Thresholds
|
||||
|
||||
```bash
|
||||
LOG_MIN_SIZE_MB=10 # skip system log if under this — keep recent history
|
||||
LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this
|
||||
|
||||
LOG_FILES=(
|
||||
"/var/log/syslog"
|
||||
"/var/log/messages"
|
||||
"/var/log/dmesg"
|
||||
)
|
||||
```
|
||||
|
||||
### Why Truncation Not Deletion
|
||||
|
||||
unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
|
||||
descriptor open and valid while emptying content — syslogd continues writing to
|
||||
the same fd without interruption. Deleting the file would orphan the file
|
||||
descriptor and syslog would stop writing until restarted.
|
||||
|
||||
### Identifying Large Docker Logs
|
||||
|
||||
```bash
|
||||
clear_logs.sh --status
|
||||
# Shows top 10 Docker logs by size, current size vs threshold
|
||||
|
||||
# Find the biggest log manually:
|
||||
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## mover_stop.sh
|
||||
|
||||
### Stop Sequence
|
||||
|
||||
```
|
||||
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
|
||||
2. Wall message to all logged-in terminal users
|
||||
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
|
||||
4. SIGTERM — mover finishes its current file operation, then stops
|
||||
5. Wait 5 seconds → verify stopped
|
||||
6. SIGKILL if still running — forced stop, partial files possible
|
||||
7. Final verify — error if still running after SIGKILL
|
||||
```
|
||||
|
||||
SIGTERM first because the mover can finish the file it is currently moving,
|
||||
leaving no partial copies split across cache and array. SIGKILL is a last resort.
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
mover_stop.sh # check and stop if running
|
||||
mover_stop.sh --status # show current mover state and PID
|
||||
mover_stop.sh --dry-run # show what would happen without stopping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## rsync_stop.sh
|
||||
|
||||
### Auto-Detection Logic
|
||||
|
||||
rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
|
||||
is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.
|
||||
|
||||
**Default behavior (orchestrator detected):**
|
||||
Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
|
||||
next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.
|
||||
|
||||
**Default behavior (no orchestrator):**
|
||||
Kills rsync directly (standalone rsync.sh run).
|
||||
|
||||
**--full-stop:**
|
||||
Kills the orchestrator first, then kills rsync. Nothing continues after this.
|
||||
Use when everything needs to stop immediately.
|
||||
|
||||
### Container Recovery
|
||||
|
||||
After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
|
||||
for any that were stopped by the interrupted rsync session and restarts them.
|
||||
Remote containers are left for docker_watchdog.sh to recover.
|
||||
|
||||
Skip container recovery with `--rsync-only` — used when called by other scripts
|
||||
that handle recovery themselves.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # smart stop (auto-detect orchestrator)
|
||||
rsync_stop.sh --full-stop # kill orchestrator + rsync
|
||||
rsync_stop.sh --rsync-only # kill rsync, skip container recovery
|
||||
rsync_stop.sh --status # show local and remote rsync state
|
||||
rsync_stop.sh --dry-run # preview without changes
|
||||
rsync_stop.sh --full-stop --dry-run # preview full stop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## user_scripts_stop.sh
|
||||
|
||||
### Process Identification
|
||||
|
||||
Scans `/proc/*/cmdline` for any process whose command line contains
|
||||
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
|
||||
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
|
||||
what the script is named or how it was launched.
|
||||
|
||||
Script names are extracted from the path for display: you see which scripts are
|
||||
being stopped, not just PIDs.
|
||||
|
||||
### Self-Exclusion
|
||||
|
||||
If this script is run via the User Scripts plugin, it would find its own PID in
|
||||
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
|
||||
from the kill list.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all User Script processes
|
||||
user_scripts_stop.sh --status # show running scripts with names and elapsed time
|
||||
user_scripts_stop.sh --dry-run # show what would be stopped
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## server_reboot.sh
|
||||
|
||||
### Full Shutdown Sequence
|
||||
|
||||
```
|
||||
1. Pre-flight checks (warn only — do not block):
|
||||
- rsync running? → warn, suggest rsync_stop.sh first
|
||||
- mover running? → warn, suggest mover_stop.sh first
|
||||
- Emby sessions? → warn (active streams will be interrupted)
|
||||
|
||||
2. Wall message to all logged-in terminal users
|
||||
|
||||
3. unRAID dashboard notification
|
||||
|
||||
4. Wait REBOOT_SLEEP seconds (default: 30)
|
||||
|
||||
5. Graceful VM shutdown:
|
||||
virsh shutdown <each VM> (ACPI signal — clean shutdown)
|
||||
Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond
|
||||
|
||||
6. /etc/rc.d/rc.libvirt stop (VM Manager)
|
||||
|
||||
7. Orchestrators/array_stopping.sh — safe ordered array stop:
|
||||
- user_scripts_stop.sh stop background User Scripts
|
||||
- fallback.sh --stop graceful fallback teardown
|
||||
- rsync_stop.sh --rsync-only kill active rsync transfers
|
||||
- mover_stop.sh stop mover
|
||||
- docker_container_stop.sh stop all containers gracefully
|
||||
|
||||
8. sync (flush filesystem buffers to disk)
|
||||
|
||||
9. /sbin/reboot
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
REBOOT_SLEEP=30 # seconds between warning and shutdown sequence
|
||||
REBOOT_VM_WAIT=30 # seconds to wait for VMs to shut down gracefully
|
||||
```
|
||||
|
||||
### Recommended Pre-Reboot Sequence
|
||||
|
||||
For a clean reboot when services are active:
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # stop any active rsync (smart mode)
|
||||
mover_stop.sh # stop mover gracefully
|
||||
server_reboot.sh --status # check what's still running
|
||||
server_reboot.sh --reason="planned maintenance"
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
server_reboot.sh # reboot with 30s warning
|
||||
server_reboot.sh --dry-run # walk through without rebooting
|
||||
server_reboot.sh --status # show what would be affected
|
||||
server_reboot.sh --reason="disk work" # include reason in notification
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
|
||||
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
WEBGUI_NGINX_WAIT=15
|
||||
WEBGUI_PHP_WAIT=10
|
||||
WEBGUI_EMHTTP_WAIT=30
|
||||
|
||||
# ── inotify Tuning ─────────────────────────────────────────────────────────────
|
||||
INOTIFY_MAX_INSTANCES=1024
|
||||
INOTIFY_MAX_WATCHES=1048576
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768
|
||||
|
||||
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
|
||||
PHP_MAX_CHILDREN=250
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
|
||||
# ── Syslog Filter ──────────────────────────────────────────────────────────────
|
||||
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
||||
|
||||
# ── Log Cleaner ────────────────────────────────────────────────────────────────
|
||||
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
|
||||
LOG_MIN_SIZE_MB=10
|
||||
LOG_DOCKER_MAX_MB=100
|
||||
|
||||
# ── Mover Stop ─────────────────────────────────────────────────────────────────
|
||||
MOVER_STOP_TIMEOUT=30
|
||||
|
||||
# ── Server Reboot ──────────────────────────────────────────────────────────────
|
||||
REBOOT_SLEEP=30
|
||||
REBOOT_VM_WAIT=30
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
|
||||
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
### rsync_stop Killed the Wrong Thing
|
||||
|
||||
If --full-stop killed an orchestrator you didn't intend to kill:
|
||||
|
||||
```bash
|
||||
# Next time use default mode (no --full-stop) to kill only rsync subprocess.
|
||||
# To verify what would be killed before running:
|
||||
rsync_stop.sh --status # shows running rsync and detected orchestrators
|
||||
rsync_stop.sh --dry-run # shows smart mode decision
|
||||
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
|
||||
```
|
||||
|
||||
### WebGUI Recovery After All Three Steps Failed
|
||||
|
||||
```bash
|
||||
# Check if processes are running:
|
||||
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
|
||||
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
|
||||
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
|
||||
|
||||
# Check recent nginx errors:
|
||||
cat /var/log/nginx/error.log | tail -20
|
||||
|
||||
# Check emhttp log:
|
||||
tail -20 /var/log/syslog | grep emhttp
|
||||
|
||||
# Last resort — reboot:
|
||||
server_reboot.sh --reason="WebGUI unrecoverable"
|
||||
```
|
||||
|
||||
### PHP-FPM Config Not Found After unRAID Update
|
||||
|
||||
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
|
||||
errors with "config file not found":
|
||||
|
||||
```bash
|
||||
# Find the new config path:
|
||||
find /etc -name "www.conf" 2>/dev/null
|
||||
|
||||
# Update PHP_CONF in master.conf:
|
||||
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
|
||||
|
||||
# Verify with:
|
||||
php_fpm_max_children.sh --status
|
||||
```
|
||||
@@ -0,0 +1,160 @@
|
||||
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
|
||||
|
||||
**System-level scripts that act on the server itself — not containers,
|
||||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||||
shutdowns with proper warning sequences.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Server Getting Into Unstable States With No Recovery Path**
|
||||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||||
write log files. The server is functionally dead but still technically running.
|
||||
|
||||
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||||
threshold breaches. Last line of defense before a hard crash.
|
||||
|
||||
**WebGUI Freezing and Nobody Noticing**
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||||
container-level alert fires because this isn't a container problem — it's a web
|
||||
server problem. By the time someone notices it may have been broken for hours.
|
||||
|
||||
Fix: `webgui_watchdog.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||||
|
||||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||||
network interface for each one. Each interface generates multiple syslog entries.
|
||||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||||
|
||||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||||
starts. Applied at array start. Idempotent — silent when already correct.
|
||||
|
||||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||||
those workers saturate immediately. New requests queue behind active ones.
|
||||
|
||||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||||
|
||||
**inotify Exhaustion Producing Unexplained Failures**
|
||||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||||
and all containers share the same pool.
|
||||
|
||||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||||
|
||||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||||
array simultaneously. unRAID's deduplication layer is confused.
|
||||
|
||||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
```
|
||||
WebGUI availability webgui_watchdog.sh — nginx → php-fpm → emhttp escalation
|
||||
Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||||
clear_logs.sh — weekly log trimming
|
||||
Graceful operations mover_stop.sh — clean mover stop
|
||||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||||
user_scripts_stop.sh — stop running User Scripts
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||||
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
```
|
||||
Orchestrators/
|
||||
array_started.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||||
─────────────────────────► docker_syslog_filter.sh (second)
|
||||
─────────────────────────► php_fpm_max_children.sh
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
|
||||
Watchdogs/
|
||||
stability_watchdog.sh and resource_watchdog.sh now live here.
|
||||
See Watchdogs/README-Watchdogs.md for how they relate to each other
|
||||
and to docker_watchdog.sh and System/storage_watchdog.sh.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every minute via watchdog_orchestrator → system_watchdog |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||||
└─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
|
||||
Every minute (watchdog_orchestrator.sh in Orchestrators/):
|
||||
→ Watchdogs/resource_watchdog.sh
|
||||
→ Watchdogs/docker_watchdog.sh
|
||||
→ Watchdogs/system_watchdog.sh (thin orchestrator)
|
||||
└─ Watchdogs/System/storage_watchdog.sh
|
||||
└─ Watchdogs/System/webgui_watchdog.sh
|
||||
WebGUI OK → silent exit
|
||||
Not responding:
|
||||
Step 1: restart nginx → recheck
|
||||
Step 2: restart php-fpm → recheck
|
||||
Step 3: restart emhttp → recheck
|
||||
All failed → notify, exit 1
|
||||
└─ Watchdogs/System/network_watchdog.sh
|
||||
→ Watchdogs/stability_watchdog.sh
|
||||
(see Watchdogs/README-Watchdogs.md for full flow)
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
System logs: clear if > LOG_MIN_SIZE_MB
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
Manual operations:
|
||||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||||
```
|
||||
Executable
+292
@@ -0,0 +1,292 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Clear Logs =================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Clears system and Docker container logs to prevent rootfs fill over time.
|
||||
# Called weekly via WEEKLY_MAINTENANCE_SCRIPTS. Uses size thresholds — only
|
||||
# clears logs large enough to be worth clearing. Small logs are left intact,
|
||||
# preserving recent diagnostic context.
|
||||
#
|
||||
# System logs (LOG_FILES): cleared if size exceeds LOG_MIN_SIZE_MB.
|
||||
# Docker logs (/var/lib/docker/containers/**/*-json.log): cleared only if the
|
||||
# individual container log exceeds LOG_DOCKER_MAX_MB. Active containers (Emby,
|
||||
# SABnzbd) grow fastest — inactive containers typically remain small.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Size Thresholds, Not Blind Truncation
|
||||
# A 2MB syslog contains useful recent diagnostic history — not worth clearing.
|
||||
# A 500MB Docker log is consuming rootfs and contains mostly noise — clear it.
|
||||
# Blind truncation destroys diagnostic context for no benefit.
|
||||
#
|
||||
# Truncation, Not Logrotate
|
||||
# unRAID writes logs to tmpfs (/var/log). Logrotate's compress + archive approach
|
||||
# would consume more tmpfs space, not less. Truncation (`: > file`) keeps the
|
||||
# file descriptor open and valid while emptying content — syslogd continues
|
||||
# writing to the same fd without interruption.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs from corrupting logs.
|
||||
#
|
||||
# Root Required
|
||||
# Truncating system logs requires root.
|
||||
#
|
||||
# Size Thresholds
|
||||
# Each file checked against its threshold before clearing.
|
||||
#
|
||||
# Silent When Clean
|
||||
# All logs below threshold = no visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# LOG_FILES
|
||||
# System log paths to check and clear. (default: /var/log/syslog /var/log/messages /var/log/dmesg)
|
||||
#
|
||||
# LOG_MIN_SIZE_MB
|
||||
# Skip system log if under this size — keep recent history. (default: 10)
|
||||
#
|
||||
# LOG_DOCKER_MAX_MB
|
||||
# Clear Docker container log only if over this size. (default: 100)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# clear_logs.sh
|
||||
# Check all configured logs. Clear those above threshold. Silent when all are small.
|
||||
#
|
||||
# clear_logs.sh --dry-run
|
||||
# Show which logs would be cleared and their current sizes. No clearing.
|
||||
#
|
||||
# clear_logs.sh --status
|
||||
# Show current log sizes vs thresholds.
|
||||
#
|
||||
# clear_logs.sh --log
|
||||
# Verbose output — show each file evaluated, its size, and action taken.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — truncating system logs requires root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be cleared"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY LOG STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Min system log: ${LOG_MIN_SIZE_MB:-10}MB before clearing"
|
||||
echo "$ICON_GEAR Max Docker log: ${LOG_DOCKER_MAX_MB:-100}MB before clearing"
|
||||
echo ""
|
||||
|
||||
echo "━━━ System Logs ━━━"
|
||||
for f in "${LOG_FILES[@]}"; do
|
||||
if [[ -f "$f" ]]; then
|
||||
size=$(du -sh "$f" 2>/dev/null | cut -f1)
|
||||
size_mb=$(du -sm "$f" 2>/dev/null | cut -f1)
|
||||
threshold="${LOG_MIN_SIZE_MB:-10}"
|
||||
if [[ "${size_mb:-0}" -ge "$threshold" ]]; then
|
||||
echo " $ICON_WARN $f — $size (above ${threshold}MB threshold — would clear)"
|
||||
else
|
||||
echo " $ICON_SUCCESS $f — $size (under threshold)"
|
||||
fi
|
||||
else
|
||||
echo " $ICON_SKIP $f — not found"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "━━━ Docker Logs (top 10 by size) ━━━"
|
||||
if [[ -d /var/lib/docker/containers ]]; then
|
||||
find /var/lib/docker/containers/ -name "*-json.log" 2>/dev/null | \
|
||||
while IFS= read -r logfile; do
|
||||
size_mb=$(du -sm "$logfile" 2>/dev/null | cut -f1)
|
||||
container_id=$(basename "$(dirname "$logfile")" | cut -c1-12)
|
||||
container_name=$(docker inspect --format '{{.Name}}' "$container_id" \
|
||||
2>/dev/null | tr -d '/' || echo "$container_id")
|
||||
echo "${size_mb:-0} $container_name $logfile"
|
||||
done | sort -rn | head -10 | \
|
||||
while read -r size_mb name logfile; do
|
||||
threshold="${LOG_DOCKER_MAX_MB:-100}"
|
||||
if [[ "$size_mb" -ge "$threshold" ]]; then
|
||||
echo " $ICON_WARN ${size_mb}MB — $name (above ${threshold}MB — would clear)"
|
||||
else
|
||||
echo " $ICON_SUCCESS ${size_mb}MB — $name"
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo " Docker directory not found"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Clear Logs ━━━
|
||||
# ==============================================================================================
|
||||
log "$ICON_GEAR Config: system-threshold=${LOG_MIN_SIZE_MB:-10}MB docker-threshold=${LOG_DOCKER_MAX_MB:-100}MB"
|
||||
log "$ICON_GEAR System logs: ${LOG_FILES[*]}"
|
||||
|
||||
START=$(date +%s)
|
||||
SYS_CLEARED=0
|
||||
SYS_SKIPPED=0
|
||||
SYS_BYTES=0
|
||||
DOCKER_CLEARED=0
|
||||
DOCKER_SKIPPED=0
|
||||
DOCKER_BYTES=0
|
||||
FAILED=()
|
||||
|
||||
# ── System Logs ───────────────────────────────────────────────────────────────────────────────
|
||||
for logfile in "${LOG_FILES[@]}"; do
|
||||
if [[ ! -f "$logfile" ]]; then
|
||||
log "$logfile — not found, skipping"
|
||||
continue
|
||||
fi
|
||||
|
||||
size_bytes=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
|
||||
size_mb=$(( size_bytes / 1048576 ))
|
||||
size_h=$(du -sh "$logfile" 2>/dev/null | cut -f1)
|
||||
threshold="${LOG_MIN_SIZE_MB:-10}"
|
||||
|
||||
if [[ "$size_mb" -lt "$threshold" ]]; then
|
||||
log "$logfile — ${size_h} (under ${threshold}MB — skipping)"
|
||||
(( SYS_SKIPPED++ ))
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would clear: $logfile (${size_h})"
|
||||
(( SYS_CLEARED++ ))
|
||||
SYS_BYTES=$(( SYS_BYTES + size_bytes ))
|
||||
continue
|
||||
fi
|
||||
|
||||
if : > "$logfile" 2>/dev/null; then
|
||||
log "Cleared: $logfile (freed ${size_h})"
|
||||
(( SYS_CLEARED++ ))
|
||||
SYS_BYTES=$(( SYS_BYTES + size_bytes ))
|
||||
else
|
||||
error "Failed to clear: $logfile"
|
||||
FAILED+=("$logfile")
|
||||
fi
|
||||
done
|
||||
|
||||
# ── Docker Logs ───────────────────────────────────────────────────────────────────────────────
|
||||
if [[ ! -d /var/lib/docker/containers ]]; then
|
||||
log "Docker containers directory not found — skipping Docker log clear"
|
||||
else
|
||||
while IFS= read -r logfile; do
|
||||
[[ -z "$logfile" ]] && continue
|
||||
|
||||
size_bytes=$(stat -c%s "$logfile" 2>/dev/null || echo 0)
|
||||
size_mb=$(( size_bytes / 1048576 ))
|
||||
size_h=$(du -sh "$logfile" 2>/dev/null | cut -f1)
|
||||
threshold="${LOG_DOCKER_MAX_MB:-100}"
|
||||
|
||||
# Get container name for display
|
||||
container_id=$(basename "$(dirname "$logfile")" | cut -c1-12)
|
||||
container_name=$(docker inspect --format '{{.Name}}' "$container_id" \
|
||||
2>/dev/null | tr -d '/' || echo "$container_id")
|
||||
|
||||
if [[ "$size_mb" -lt "$threshold" ]]; then
|
||||
log "Docker $container_name — ${size_h} (under ${threshold}MB — skipping)"
|
||||
(( DOCKER_SKIPPED++ ))
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would clear Docker log: $container_name (${size_h})"
|
||||
(( DOCKER_CLEARED++ ))
|
||||
DOCKER_BYTES=$(( DOCKER_BYTES + size_bytes ))
|
||||
continue
|
||||
fi
|
||||
|
||||
if : > "$logfile" 2>/dev/null; then
|
||||
log "Cleared Docker log: $container_name (freed ${size_h})"
|
||||
(( DOCKER_CLEARED++ ))
|
||||
DOCKER_BYTES=$(( DOCKER_BYTES + size_bytes ))
|
||||
else
|
||||
error "Failed to clear Docker log: $container_name"
|
||||
FAILED+=("docker:$container_name")
|
||||
fi
|
||||
done < <(find /var/lib/docker/containers/ -name "*-json.log" 2>/dev/null)
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
TOTAL_BYTES=$(( SYS_BYTES + DOCKER_BYTES ))
|
||||
TOTAL_FREED_H=$(awk "BEGIN {printf \"%.1fMB\", $TOTAL_BYTES / 1048576}")
|
||||
TOTAL_CLEARED=$(( SYS_CLEARED + DOCKER_CLEARED ))
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$TOTAL_CLEARED" -gt 0 || ${#FAILED[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY LOG CLEANER SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_HEALTH System cleared: $SYS_CLEARED file(s)"
|
||||
echo "$ICON_CONTAINERS Docker cleared: $DOCKER_CLEARED file(s)"
|
||||
echo "$ICON_HEALTH Total freed: $TOTAL_FREED_H"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
[[ "$SYS_SKIPPED" -gt 0 || "$DOCKER_SKIPPED" -gt 0 ]] && \
|
||||
log "Skipped: ${SYS_SKIPPED} system + ${DOCKER_SKIPPED} Docker (under threshold)"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no files cleared"
|
||||
elif [[ ${#FAILED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_ERROR Status: SOME FILES FAILED — ${FAILED[*]}"
|
||||
notify "Log clear failed on $(hostname) ($MY_ID) — ${FAILED[*]}" \
|
||||
"Clear Logs" "warning"
|
||||
else
|
||||
echo "$ICON_DONE Status: done — ${TOTAL_FREED_H} freed"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
# All logs under threshold — completely silent
|
||||
echo "All logs under threshold — nothing to clear"
|
||||
fi
|
||||
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && exit 1
|
||||
exit 0
|
||||
Executable
+138
@@ -0,0 +1,138 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Conf Cache Sync ================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Maintains a RAM-resident conf cache at /tmp/.vv/config/cached/.confs/.
|
||||
# Credentials and partner keys live in RAM only — never on disk across hosts.
|
||||
#
|
||||
# On array start (default / --array-start):
|
||||
# 1. Copy own conf to local cache
|
||||
# 2. Pull each available partner's conf from their disk → local cache
|
||||
# 3. Push own conf to each available partner's /tmp/.vv/ cache
|
||||
#
|
||||
# On conf save (--push-only):
|
||||
# Fast path — push updated own conf to all partners' /tmp/.vv/ cache only.
|
||||
# No pulls, no local cache rebuild.
|
||||
#
|
||||
# Cache is /tmp (tmpfs) — cleared every reboot, repopulated by this script
|
||||
# on next array start. Scripts source from cache for partner vars; own vars
|
||||
# always come from disk (load_config.sh skips cached copy of own conf).
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# conf_sync.sh Full sync: pull from all partners + push to all partners
|
||||
# conf_sync.sh --push-only Push own conf to all partners (fast, for conf-save hook)
|
||||
# conf_sync.sh --dry-run Show what would happen, no changes
|
||||
# conf_sync.sh --log Verbose output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
PUSH_ONLY=false
|
||||
FILTERED_ARGS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--push-only) PUSH_ONLY=true ;;
|
||||
*) FILTERED_ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
detect_hosts
|
||||
|
||||
CACHE_DIR="/tmp/.vv/config/cached/.confs"
|
||||
MY_CONF="$SCRIPTS_ROOT/Configurations/${MY_ID,,}.conf"
|
||||
SSH_TIMEOUT=10
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ── Ensure cache dir exists ───────────────────────────────────────────────────
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
mkdir -p "$CACHE_DIR"
|
||||
fi
|
||||
|
||||
# ── Copy own conf into local cache ───────────────────────────────────────────
|
||||
if [[ "$PUSH_ONLY" == false ]]; then
|
||||
if [[ -f "$MY_CONF" ]]; then
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would copy $(basename "$MY_CONF") → $CACHE_DIR/"
|
||||
else
|
||||
cp "$MY_CONF" "$CACHE_DIR/${MY_ID,,}.conf" && \
|
||||
log "Own conf cached ✅" || warn "Failed to cache own conf"
|
||||
fi
|
||||
else
|
||||
warn "Own conf not found: $MY_CONF"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Per-partner sync ──────────────────────────────────────────────────────────
|
||||
PUSHED=0
|
||||
PULLED=0
|
||||
FAILED=0
|
||||
|
||||
for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
|
||||
partner_host="${!host_var}"
|
||||
[[ -z "$partner_host" ]] && continue
|
||||
[[ "${host_var,,}" == "${MY_ID,,}" ]] && continue
|
||||
|
||||
partner_slot="${host_var,,}" # e.g. host2
|
||||
partner_ip=$(resolve_tailscale_ip "$partner_host" 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$partner_ip" ]]; then
|
||||
warn "$partner_host — cannot resolve Tailscale IP, skipping"
|
||||
(( FAILED++ ))
|
||||
continue
|
||||
fi
|
||||
|
||||
# ── Pull: grab partner's conf from their disk → our local cache ──────────
|
||||
if [[ "$PUSH_ONLY" == false ]]; then
|
||||
remote_conf="/boot/config/plugins/varaverk/Configurations/${partner_slot}.conf"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull $partner_host:$remote_conf → $CACHE_DIR/${partner_slot}.conf"
|
||||
elif timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
|
||||
"root@${partner_ip}:${remote_conf}" \
|
||||
"$CACHE_DIR/${partner_slot}.conf" 2>/dev/null; then
|
||||
log "Pulled ${partner_slot}.conf from $partner_host ✅"
|
||||
(( PULLED++ ))
|
||||
else
|
||||
warn "Could not pull ${partner_slot}.conf from $partner_host"
|
||||
(( FAILED++ ))
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Push: send own conf to partner's /tmp/.vv/ cache ────────────────────
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would push ${MY_ID,,}.conf → $partner_host:/tmp/.vv/config/cached/.confs/"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Ensure partner's cache dir exists, then SCP own conf into it
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
|
||||
"root@${partner_ip}" "mkdir -p '$CACHE_DIR'" 2>/dev/null
|
||||
|
||||
if timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
|
||||
"$MY_CONF" \
|
||||
"root@${partner_ip}:${CACHE_DIR}/${MY_ID,,}.conf" 2>/dev/null; then
|
||||
log "Pushed ${MY_ID,,}.conf to $partner_host ✅"
|
||||
(( PUSHED++ ))
|
||||
else
|
||||
warn "Could not push to $partner_host"
|
||||
(( FAILED++ ))
|
||||
fi
|
||||
done
|
||||
|
||||
# ── Summary ───────────────────────────────────────────────────────────────────
|
||||
if [[ "$PUSH_ONLY" == true ]]; then
|
||||
info "Conf push complete — pushed to $PUSHED host(s)${FAILED:+, $FAILED failed}"
|
||||
else
|
||||
info "Conf sync complete — pulled $PULLED, pushed $PUSHED${FAILED:+, $FAILED failed}"
|
||||
fi
|
||||
Executable
+230
@@ -0,0 +1,230 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Docker Syslog Filter ===========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Suppresses Docker veth/docker0 network interface messages from syslog. Run
|
||||
# once at array start via ARRAY_START_SCRIPTS before any containers start.
|
||||
# Idempotent — completely silent when the filter is already in place.
|
||||
#
|
||||
# Every container start/stop generates kernel messages like:
|
||||
# "veth2a3b4c5: renamed from eth0"
|
||||
# "docker0: port 1(veth...) entered forwarding state"
|
||||
# 50+ containers at array start = 200–400 lines of noise in the first minute,
|
||||
# completely masking real events.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Idempotent Content Check
|
||||
# Checks whether the filter file already exists with exactly the expected content.
|
||||
# If already correct → silent exit, no rsyslog restart. Only writes and restarts
|
||||
# rsyslog when the filter is missing or content has changed. Running at every boot
|
||||
# without this would cause an unnecessary rsyslog restart on every array start.
|
||||
#
|
||||
# rsyslog Drop Rule
|
||||
# Creates FILTER_FILE (/etc/rsyslog.d/ignore-docker-veth.conf).
|
||||
# Uses RainerScript `stop` action — messages matching "veth" or "docker0" are
|
||||
# dropped before reaching any output target, including the log file.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# Writing to /etc/rsyslog.d/ requires root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs at array start.
|
||||
#
|
||||
# rsyslog Process Verify
|
||||
# Confirms rsyslog is running after restart.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs every boot — no noise when filter is already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# FILTER_FILE
|
||||
# Path for the rsyslog drop filter config file.
|
||||
# (default: /etc/rsyslog.d/ignore-docker-veth.conf)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_syslog_filter.sh
|
||||
# Check filter file. Write and restart rsyslog only if changed. Silent if correct.
|
||||
#
|
||||
# docker_syslog_filter.sh --dry-run
|
||||
# Show what would be written without writing or restarting rsyslog.
|
||||
#
|
||||
# docker_syslog_filter.sh --status
|
||||
# Show current filter file content and rsyslog process state.
|
||||
#
|
||||
# docker_syslog_filter.sh --log
|
||||
# Verbose output showing the content comparison and any changes applied.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# Expected filter content — used for idempotent check
|
||||
EXPECTED_FILTER='if ($msg contains "veth" or $msg contains "docker0") then {
|
||||
stop
|
||||
}'
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — writing to /etc/rsyslog.d/ requires root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SYSLOG FILTER STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_HEALTH Filter file: $FILTER_FILE"
|
||||
echo ""
|
||||
|
||||
if [[ -f "$FILTER_FILE" ]]; then
|
||||
echo " Filter file: EXISTS"
|
||||
echo ""
|
||||
echo " Current content:"
|
||||
while IFS= read -r line; do
|
||||
echo " $line"
|
||||
done < "$FILTER_FILE"
|
||||
echo ""
|
||||
|
||||
if [[ "$(cat "$FILTER_FILE" 2>/dev/null)" == "$EXPECTED_FILTER" ]]; then
|
||||
echo " $ICON_SUCCESS Content: correct ✅"
|
||||
else
|
||||
echo " $ICON_WARN Content: differs from expected — would be rewritten"
|
||||
fi
|
||||
else
|
||||
echo " Filter file: NOT FOUND — would be created"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━ rsyslog Status ━━━"
|
||||
if platform_is_service_running rsyslog; then
|
||||
echo " rsyslogd: running ✅"
|
||||
else
|
||||
echo " rsyslogd: NOT running"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Idempotent Check ━━━
|
||||
# ==============================================================================================
|
||||
if [[ -f "$FILTER_FILE" ]] && \
|
||||
[[ "$(cat "$FILTER_FILE" 2>/dev/null)" == "$EXPECTED_FILTER" ]]; then
|
||||
echo "Syslog filter already correct ✅"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Apply Filter ━━━
|
||||
# ==============================================================================================
|
||||
START=$(date +%s)
|
||||
|
||||
log "Filter file missing or outdated — applying..."
|
||||
|
||||
# Ensure rsyslog.d directory exists
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
mkdir -p "$(dirname "$FILTER_FILE")" || {
|
||||
error "Failed to create directory: $(dirname "$FILTER_FILE")"
|
||||
exit 1
|
||||
}
|
||||
fi
|
||||
|
||||
# Write filter file
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would write filter to: $FILTER_FILE"
|
||||
warn "Content:"
|
||||
echo "$EXPECTED_FILTER" | while IFS= read -r line; do
|
||||
echo " $line"
|
||||
done
|
||||
else
|
||||
echo "$EXPECTED_FILTER" > "$FILTER_FILE" || {
|
||||
error "Failed to write filter file: $FILTER_FILE"
|
||||
notify "Syslog filter write failed on $(hostname) ($MY_ID)" \
|
||||
"Syslog Filter" "warning"
|
||||
exit 1
|
||||
}
|
||||
log "Filter file written: $FILTER_FILE"
|
||||
fi
|
||||
|
||||
# Restart rsyslog to apply
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart rsyslog"
|
||||
else
|
||||
echo "Restarting rsyslog..."
|
||||
if platform_restart_service rsyslog; then
|
||||
sleep 2
|
||||
# Verify rsyslog actually running after restart
|
||||
if platform_is_service_running rsyslog; then
|
||||
echo "rsyslog restarted and running ✅"
|
||||
else
|
||||
error "rsyslog not running after restart"
|
||||
notify "rsyslog failed to start after filter update on $(hostname) ($MY_ID)" \
|
||||
"Syslog Filter" "warning"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
error "rsyslog restart command failed"
|
||||
notify "rsyslog restart failed on $(hostname) ($MY_ID) — filter may not be active" \
|
||||
"Syslog Filter" "warning"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SYSLOG FILTER SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_HEALTH Filter file: $FILTER_FILE"
|
||||
echo "$ICON_HEALTH Targets: veth / docker0"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
echo "$ICON_DONE Status: done — Docker veth noise suppressed ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
exit 0
|
||||
Executable
+240
@@ -0,0 +1,240 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= inotify Tuning ============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Raises Linux inotify limits at array start to prevent exhaustion across the
|
||||
# container stack. Run once at array start via ARRAY_START_SCRIPTS. Settings
|
||||
# are lost on reboot — this script reapplies them on every array start.
|
||||
#
|
||||
# Must run FIRST in ARRAY_START_SCRIPTS before any containers start — containers
|
||||
# inherit inotify limits at launch, not dynamically. Running this after Code-Server
|
||||
# starts requires a docker restart to pick up the new values.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three inotify Limits
|
||||
# max_user_instances — max independent inotify fd objects per user; each container
|
||||
# calling inotify_init() consumes one. Default 128 — exhausted
|
||||
# quickly with 20+ active containers.
|
||||
#
|
||||
# max_user_watches — SHARED budget across ALL users and containers on the system.
|
||||
# Each watched file or directory costs one watch. Default 8192 —
|
||||
# VSCode alone needs 50K–200K for large workspaces. Combined
|
||||
# usage of Sonarr, Radarr, Lidarr, Emby, Nextcloud, Code-Server
|
||||
# easily exceeds 512K on a busy server.
|
||||
#
|
||||
# max_queued_events — max events buffered before the kernel drops them. Low value =
|
||||
# events silently lost during high-activity bursts. Default 16384.
|
||||
#
|
||||
# Why 1M Watches
|
||||
# Raising max_user_watches to 1048576 (1M) gives sufficient headroom for all
|
||||
# containers combined — safe on 128GB RAM (~128MB kernel use for the pool).
|
||||
#
|
||||
# Idempotent Per-Setting
|
||||
# Each sysctl value is read before writing. Only changed if different from target —
|
||||
# no-op on boots where limits are already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# sysctl writes require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate runs at array start.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs every boot — no noise when already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# INOTIFY_MAX_INSTANCES
|
||||
# Max inotify fd objects per user. (default: 1024)
|
||||
#
|
||||
# INOTIFY_MAX_WATCHES
|
||||
# Max watched files/dirs shared across all users and containers. (default: 1048576)
|
||||
#
|
||||
# INOTIFY_MAX_QUEUED_EVENTS
|
||||
# Max events buffered before kernel drops them. (default: 32768)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# inotify_tuning.sh
|
||||
# Apply inotify limits. No-op per setting if already at target.
|
||||
#
|
||||
# inotify_tuning.sh --dry-run
|
||||
# Show current vs target for each limit. No sysctl writes.
|
||||
#
|
||||
# inotify_tuning.sh --status
|
||||
# Show current vs target, active instance count, and top consumers by PID.
|
||||
#
|
||||
# inotify_tuning.sh --log
|
||||
# Verbose output — show each sysctl check and result.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — sysctl writes require root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY INOTIFY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
|
||||
echo "━━━ Kernel Limits ━━━"
|
||||
CURRENT_INSTANCES=$(sysctl -n fs.inotify.max_user_instances 2>/dev/null || echo "?")
|
||||
CURRENT_WATCHES=$(sysctl -n fs.inotify.max_user_watches 2>/dev/null || echo "?")
|
||||
CURRENT_EVENTS=$(sysctl -n fs.inotify.max_queued_events 2>/dev/null || echo "?")
|
||||
|
||||
for label in "max_user_instances current=$CURRENT_INSTANCES target=$INOTIFY_MAX_INSTANCES" \
|
||||
"max_user_watches current=$CURRENT_WATCHES target=$INOTIFY_MAX_WATCHES" \
|
||||
"max_queued_events current=$CURRENT_EVENTS target=$INOTIFY_MAX_QUEUED_EVENTS"; do
|
||||
echo " $label"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "━━━ Active Instances ━━━"
|
||||
USED_INSTANCES=$(find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l)
|
||||
USED_INSTANCES="${USED_INSTANCES//[^0-9]/}"
|
||||
echo " Instances in use: ${USED_INSTANCES:-0} / $CURRENT_INSTANCES"
|
||||
if [[ "$CURRENT_INSTANCES" -gt 0 ]]; then
|
||||
PCT=$(( ${USED_INSTANCES:-0} * 100 / CURRENT_INSTANCES ))
|
||||
echo " Utilisation: ${PCT}%"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━ Top Consumers ━━━"
|
||||
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | \
|
||||
awk -F/ '{print $3}' | sort | uniq -c | sort -rn | head -10 | \
|
||||
while read -r count pid; do
|
||||
cmd=$(cat /proc/"$pid"/comm 2>/dev/null || echo "?")
|
||||
cgroup=$(cat /proc/"$pid"/cgroup 2>/dev/null | \
|
||||
grep docker | grep -o '[a-f0-9]\{12\}' | head -1 || echo "")
|
||||
if [[ -n "$cgroup" ]]; then
|
||||
label="[docker:${cgroup}] $cmd"
|
||||
else
|
||||
label="[host] $cmd"
|
||||
fi
|
||||
echo " ${count} instances — $label (PID $pid)"
|
||||
done | head -10
|
||||
|
||||
echo ""
|
||||
echo "━━━ VSCode / Code-Server ━━━"
|
||||
echo " If VSCode shows 'unable to watch for file changes':"
|
||||
echo " 1. Verify max_user_watches target is set high enough"
|
||||
echo " 2. Check total watches used: cat /proc/sys/fs/inotify/max_user_watches"
|
||||
echo " 3. After any limit change: docker restart Code-Server"
|
||||
echo " (running containers inherit limits at start, not dynamically)"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Apply Settings ━━━
|
||||
# ==============================================================================================
|
||||
CHANGED=0
|
||||
FAILED=0
|
||||
|
||||
apply_sysctl() {
|
||||
local key="$1" value="$2"
|
||||
local current
|
||||
current=$(sysctl -n "$key" 2>/dev/null || echo 0)
|
||||
|
||||
if [[ "$current" -eq "$value" ]]; then
|
||||
log "$key = $value (already correct)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would set $key = $value (currently $current)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
if sysctl -w "${key}=${value}" >/dev/null 2>&1; then
|
||||
warn "Set $key = $value (was $current)"
|
||||
(( CHANGED++ ))
|
||||
else
|
||||
error "Failed to set $key = $value"
|
||||
(( FAILED++ ))
|
||||
fi
|
||||
}
|
||||
|
||||
apply_sysctl "fs.inotify.max_user_instances" "$INOTIFY_MAX_INSTANCES"
|
||||
apply_sysctl "fs.inotify.max_user_watches" "$INOTIFY_MAX_WATCHES"
|
||||
apply_sysctl "fs.inotify.max_queued_events" "$INOTIFY_MAX_QUEUED_EVENTS"
|
||||
apply_sysctl "vm.overcommit_memory" "1" # Redis: prevents background save failures under low memory
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$FAILED" -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY INOTIFY TUNING SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_ERROR $FAILED setting(s) failed to apply"
|
||||
notify "inotify tuning failed on $(hostname) ($MY_ID) — $FAILED setting(s) could not be applied" \
|
||||
"inotify Tuning" "warning"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 1
|
||||
elif [[ "$CHANGED" -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY INOTIFY TUNING SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo " max_user_instances: $(sysctl -n fs.inotify.max_user_instances 2>/dev/null)"
|
||||
echo " max_user_watches: $(sysctl -n fs.inotify.max_user_watches 2>/dev/null)"
|
||||
echo " max_queued_events: $(sysctl -n fs.inotify.max_queued_events 2>/dev/null)"
|
||||
echo ""
|
||||
warn "$CHANGED setting(s) updated"
|
||||
if [[ "$CHANGED" -gt 0 ]]; then
|
||||
warn "If Code-Server is running: docker restart Code-Server"
|
||||
warn "Running containers inherit limits at start — restart picks up new values"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
echo "inotify limits already correct ✅"
|
||||
fi
|
||||
|
||||
exit 0
|
||||
Executable
+195
@@ -0,0 +1,195 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Mover Stop =================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Safely stops the unRAID mover with a wall warning, configurable timeout, and
|
||||
# SIGTERM → SIGKILL sequence. Use before planned reboots, disk operations, or
|
||||
# any operation where mover and rsync running simultaneously could corrupt files.
|
||||
# Exits cleanly if mover is not running.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Stop Sequence
|
||||
# 1. Check if mover is running — exit cleanly if not
|
||||
# 2. Wall message to all logged-in terminal users
|
||||
# 3. Wait MOVER_STOP_TIMEOUT seconds
|
||||
# 4. SIGTERM — allows mover to finish its current file before stopping
|
||||
# (no partial files — the mover completes what it is working on)
|
||||
# 5. Wait 5 seconds → verify stopped
|
||||
# 6. SIGKILL if still running — forced stop, partial files possible
|
||||
# 7. Final verify — error if still running after SIGKILL
|
||||
#
|
||||
# SIGTERM first because the mover has an opportunity to finish the file it is
|
||||
# currently moving, leaving no partial copies on cache or array. SIGKILL is only
|
||||
# used as a last resort and may leave a file split across cache and array.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts racing each other.
|
||||
#
|
||||
# Root Required
|
||||
# pkill on emhttp processes requires root.
|
||||
#
|
||||
# Final Verify
|
||||
# Confirms mover is actually stopped after the kill sequence — errors if it
|
||||
# is still running after SIGKILL.
|
||||
#
|
||||
# Silent When Clean
|
||||
# Mover not running = log() only, no visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# MOVER_STOP_TIMEOUT
|
||||
# Seconds between wall warning and SIGTERM. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# mover_stop.sh
|
||||
# Check if mover is running. If so, warn users and stop it.
|
||||
#
|
||||
# mover_stop.sh --dry-run
|
||||
# Show mover state and what would happen. No signals sent.
|
||||
#
|
||||
# mover_stop.sh --status
|
||||
# Show mover state (running, PID, start time). Then exit.
|
||||
#
|
||||
# mover_stop.sh --log
|
||||
# Verbose output showing each step of the stop sequence.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — pkill on emhttp processes requires root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
validate_int MOVER_STOP_TIMEOUT "$MOVER_STOP_TIMEOUT"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_MOVER Timeout: ${MOVER_STOP_TIMEOUT}s"
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo ""
|
||||
if platform_is_mover_running; then
|
||||
MOVER_PID=$(platform_get_mover_pid)
|
||||
MOVER_START=$(ps -o lstart= -p "$MOVER_PID" 2>/dev/null | xargs)
|
||||
echo " $ICON_MOVER Mover: RUNNING (PID $MOVER_PID)"
|
||||
[[ -n "$MOVER_START" ]] && echo " $ICON_TIME Started: $MOVER_START"
|
||||
else
|
||||
echo " $ICON_MOVER Mover: not running"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Mover Stop ━━━
|
||||
# ==============================================================================================
|
||||
START=$(date +%s)
|
||||
|
||||
if ! platform_is_mover_running; then
|
||||
echo "Mover is not running — nothing to do"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
MOVER_PID=$(platform_get_mover_pid)
|
||||
MOVER_START=$(ps -o lstart= -p "$MOVER_PID" 2>/dev/null | xargs)
|
||||
MOVER_ELAPSED=$(ps -o etimes= -p "$MOVER_PID" 2>/dev/null | tr -d ' ')
|
||||
warn "Mover is running (PID $MOVER_PID) — stopping in ${MOVER_STOP_TIMEOUT}s"
|
||||
log "$ICON_TIME Mover started: ${MOVER_START:-unknown} — running for $(format_duration "${MOVER_ELAPSED:-0}")"
|
||||
|
||||
# ── Warn users via wall ───────────────────────────────────────────────────────────────────────
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
wall "$ICON_WARN $MY_ID ($LOCAL_SERVER_NAME) — unRAID Mover stopping in ${MOVER_STOP_TIMEOUT}s"
|
||||
echo "Wall message sent — waiting ${MOVER_STOP_TIMEOUT}s..."
|
||||
sleep "$MOVER_STOP_TIMEOUT"
|
||||
else
|
||||
warn "DRY RUN — would send wall warning and wait ${MOVER_STOP_TIMEOUT}s"
|
||||
fi
|
||||
|
||||
# ── SIGTERM — graceful stop ───────────────────────────────────────────────────────────────────
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would send SIGTERM to mover (PID $MOVER_PID)"
|
||||
else
|
||||
log "Sending SIGTERM to mover (PID $MOVER_PID)..."
|
||||
kill -TERM "$MOVER_PID" 2>/dev/null || true
|
||||
sleep 5
|
||||
|
||||
# Verify stopped after SIGTERM
|
||||
if ! platform_is_mover_running; then
|
||||
warn "Mover stopped cleanly (SIGTERM) ✅"
|
||||
else
|
||||
# ── SIGKILL — forced stop ─────────────────────────────────────────────────────────────
|
||||
warn "Mover still running after SIGTERM — sending SIGKILL (may leave partial files)"
|
||||
kill -KILL "$MOVER_PID" 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
# Final verify
|
||||
if platform_is_mover_running; then
|
||||
error "Mover still running after SIGKILL — manual intervention needed"
|
||||
notify "Mover stop failed on $(hostname) ($MY_ID) — process unkillable" \
|
||||
"Mover Stop" "warning"
|
||||
exit 1
|
||||
else
|
||||
warn "Mover force-stopped (SIGKILL) — check for partial files on cache"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY MOVER STOP SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_MOVER Timeout: ${MOVER_STOP_TIMEOUT}s"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
echo "$ICON_DONE Status: done — mover stopped ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
Executable
+247
@@ -0,0 +1,247 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= PHP-FPM Max Children ===========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Raises PHP-FPM pm.max_children to prevent WebGUI slowdowns under load. Run
|
||||
# once at array start via ARRAY_START_SCRIPTS. Idempotent — silent when the
|
||||
# value is already correct, no restart on clean boot.
|
||||
#
|
||||
# unRAID's WebGUI runs through PHP-FPM. The default pm.max_children is very
|
||||
# low (4–8). Under load — multiple users, Docker operations, heavy dashboard
|
||||
# usage — all PHP workers saturate and new requests queue. The WebGUI becomes
|
||||
# slow or unresponsive.
|
||||
#
|
||||
# PHP_MAX_CHILDREN=250 is appropriate for 128GB RAM: ~2MB per worker = ~500MB
|
||||
# total. Too high wastes RAM; too low causes slowdowns.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Idempotent Content Check
|
||||
# Reads the current pm.max_children value before writing. If already at
|
||||
# target → silent exit, no PHP-FPM restart. Restarting PHP-FPM unnecessarily
|
||||
# disrupts active WebGUI sessions on every boot.
|
||||
#
|
||||
# Pattern Match Before Write
|
||||
# Verifies the sed pattern finds pm.max_children in the config before
|
||||
# applying any change. Prevents silent failures where sed succeeds but
|
||||
# writes nothing because the key was missing or commented out.
|
||||
#
|
||||
# Apply Sequence
|
||||
# 1. Read current pm.max_children from PHP_CONF
|
||||
# 2. If already at target → exit silently
|
||||
# 3. Verify sed pattern matches before writing
|
||||
# 4. Apply sed replacement
|
||||
# 5. Restart PHP-FPM via rc.php-fpm
|
||||
# 6. Verify PHP-FPM process running after restart
|
||||
# 7. Read back config to confirm value applied
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# Writing to /etc/php83/ requires root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs at array start.
|
||||
#
|
||||
# Process Verify
|
||||
# Confirms PHP-FPM running after restart — errors if it failed to start.
|
||||
#
|
||||
# Config Verify
|
||||
# Reads back config after restart to confirm the value was actually applied.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs every boot — no noise when already correct.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# PHP_MAX_CHILDREN
|
||||
# Target pm.max_children value. (default: 250)
|
||||
#
|
||||
# PHP_CONF
|
||||
# Path to PHP-FPM www.conf. (default: /etc/php83/php-fpm.d/www.conf)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# php_fpm_max_children.sh
|
||||
# Read current value. Update and restart PHP-FPM only if changed. Silent if correct.
|
||||
#
|
||||
# php_fpm_max_children.sh --dry-run
|
||||
# Show current vs target value. No config write or restart.
|
||||
#
|
||||
# php_fpm_max_children.sh --status
|
||||
# Show current pm.max_children, target, and PHP-FPM process state.
|
||||
#
|
||||
# php_fpm_max_children.sh --log
|
||||
# Verbose output showing idempotent check, config write, and restart result.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — writing system config requires root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
validate_int PHP_MAX_CHILDREN "$PHP_MAX_CHILDREN"
|
||||
require_var PHP_CONF
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY PHP-FPM STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Config file: $PHP_CONF"
|
||||
echo "$ICON_PHP Target: pm.max_children = $PHP_MAX_CHILDREN"
|
||||
echo ""
|
||||
|
||||
if [[ -f "$PHP_CONF" ]]; then
|
||||
CURRENT_VAL=$(grep -E "^pm\.max_children" "$PHP_CONF" 2>/dev/null | \
|
||||
awk '{print $NF}')
|
||||
if [[ "${CURRENT_VAL:-0}" -eq "$PHP_MAX_CHILDREN" ]]; then
|
||||
echo " $ICON_SUCCESS Current: pm.max_children = $CURRENT_VAL (correct ✅)"
|
||||
else
|
||||
echo " $ICON_WARN Current: pm.max_children = ${CURRENT_VAL:-not set} (would update)"
|
||||
fi
|
||||
else
|
||||
echo " $ICON_ERROR Config file not found: $PHP_CONF"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if pgrep -f "php-fpm" >/dev/null 2>&1; then
|
||||
FPM_COUNT=$(pgrep -fc "php-fpm" 2>/dev/null || echo "?")
|
||||
echo " $ICON_SUCCESS PHP-FPM: running ($FPM_COUNT worker(s))"
|
||||
else
|
||||
echo " $ICON_ERROR PHP-FPM: NOT running"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ PHP-FPM Config ━━━
|
||||
# ==============================================================================================
|
||||
START=$(date +%s)
|
||||
|
||||
if [[ ! -f "$PHP_CONF" ]]; then
|
||||
error "PHP config file not found: $PHP_CONF"
|
||||
notify "PHP-FPM config not found on $(hostname) ($MY_ID) — $PHP_CONF missing" \
|
||||
"PHP-FPM" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Idempotent check ─────────────────────────────────────────────────────────────────────────
|
||||
CURRENT_VAL=$(grep -E "^pm\.max_children" "$PHP_CONF" 2>/dev/null | awk '{print $NF}')
|
||||
if [[ "${CURRENT_VAL:-0}" -eq "$PHP_MAX_CHILDREN" ]]; then
|
||||
echo "pm.max_children already $PHP_MAX_CHILDREN ✅"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
warn "pm.max_children: ${CURRENT_VAL:-not set} → $PHP_MAX_CHILDREN"
|
||||
|
||||
# ── Verify pattern exists before writing ─────────────────────────────────────────────────────
|
||||
if ! grep -qE "^pm\.max_children" "$PHP_CONF" 2>/dev/null; then
|
||||
error "pm.max_children not found in $PHP_CONF — cannot apply"
|
||||
error "Add 'pm.max_children = $PHP_MAX_CHILDREN' to $PHP_CONF manually"
|
||||
notify "PHP-FPM pm.max_children not found in config on $(hostname) ($MY_ID)" \
|
||||
"PHP-FPM" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would set pm.max_children = $PHP_MAX_CHILDREN in $PHP_CONF"
|
||||
warn "DRY RUN — would restart PHP-FPM"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Apply setting ─────────────────────────────────────────────────────────────────────────────
|
||||
log "Applying pm.max_children = $PHP_MAX_CHILDREN..."
|
||||
if ! sed -i "s/^pm\.max_children.*/pm.max_children = $PHP_MAX_CHILDREN/" "$PHP_CONF"; then
|
||||
error "Failed to update $PHP_CONF"
|
||||
notify "PHP-FPM config update failed on $(hostname) ($MY_ID)" \
|
||||
"PHP-FPM" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log "Config updated"
|
||||
|
||||
# ── Restart PHP-FPM ──────────────────────────────────────────────────────────────────────────
|
||||
log "Restarting PHP-FPM..."
|
||||
if ! platform_restart_service php-fpm; then
|
||||
error "PHP-FPM restart command failed"
|
||||
notify "PHP-FPM restart failed on $(hostname) ($MY_ID)" \
|
||||
"PHP-FPM" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 3 # Allow PHP-FPM workers to initialise
|
||||
|
||||
# ── Verify process running ────────────────────────────────────────────────────────────────────
|
||||
if ! pgrep -f "php-fpm" >/dev/null 2>&1; then
|
||||
error "PHP-FPM not running after restart — WebGUI may be broken"
|
||||
notify "PHP-FPM failed to start after config update on $(hostname) ($MY_ID)" \
|
||||
"PHP-FPM" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ── Verify config reflects target ────────────────────────────────────────────────────────────
|
||||
APPLIED_VAL=$(grep -E "^pm\.max_children" "$PHP_CONF" 2>/dev/null | awk '{print $NF}')
|
||||
if [[ "${APPLIED_VAL:-0}" -ne "$PHP_MAX_CHILDREN" ]]; then
|
||||
warn "Config reads pm.max_children = ${APPLIED_VAL:-unknown} — expected $PHP_MAX_CHILDREN"
|
||||
warn "Check $PHP_CONF manually"
|
||||
else
|
||||
log "Verified: pm.max_children = $APPLIED_VAL ✅"
|
||||
fi
|
||||
|
||||
FPM_WORKERS=$(pgrep -fc "php-fpm" 2>/dev/null || echo "?")
|
||||
log "$ICON_PHP Workers running: $FPM_WORKERS"
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY PHP-FPM SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Config file: $PHP_CONF"
|
||||
echo "$ICON_PHP Applied: pm.max_children = $PHP_MAX_CHILDREN"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
exit 0
|
||||
Executable
+518
@@ -0,0 +1,518 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Rsync Stop =================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Stops rsync intelligently on both local and remote servers. Auto-detects
|
||||
# running orchestrators and chooses the safest stop strategy. If an
|
||||
# orchestrator is running, kills only the rsync subprocess so the orchestrator
|
||||
# exits cleanly after finishing the current share. Use --full-stop to kill
|
||||
# everything immediately.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Two Stop Modes
|
||||
# Default (smart):
|
||||
# Detects if an orchestrator (daily/weekly/critical sync) is running.
|
||||
# If orchestrator found → kills rsync subprocess only. Orchestrator sees
|
||||
# rsync exit → moves to next share or exits cleanly on its own.
|
||||
# If no orchestrator → kills rsync directly (standalone rsync.sh run).
|
||||
# Cleans stale lock files after kill.
|
||||
# Recovers containers left stopped by interrupted rsync (local only).
|
||||
#
|
||||
# --full-stop (nuclear):
|
||||
# Kills orchestrator first → then kills rsync.
|
||||
# Orchestrator will NOT continue to next share.
|
||||
# Use when everything needs to stop immediately.
|
||||
#
|
||||
# Orchestrator Detection
|
||||
# detect_rsync_parent() scans all lock files to find which running process
|
||||
# has rsync as a descendant. No hardcoded list — works for any orchestrator.
|
||||
# Returns "script_name:parent_pid" if found, empty if standalone.
|
||||
#
|
||||
# Remote Handling
|
||||
# Both local and remote handled in one run via SSH.
|
||||
# Remote containers left as-is — docker_watchdog.sh handles remote recovery.
|
||||
# If remote unreachable → skips remote cleanly, logs warning.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# pkill and docker require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts racing each other.
|
||||
#
|
||||
# Timeout Protection
|
||||
# DOCKER_TIMEOUT (15s) on all docker calls — hung daemon doesn't block.
|
||||
# SSH_TIMEOUT (15s) on all remote SSH calls.
|
||||
#
|
||||
# SIGTERM → SIGKILL Sequence
|
||||
# Orchestrators receive SIGTERM first, SIGKILL only if still running after 2s.
|
||||
#
|
||||
# Container Recovery
|
||||
# Restarts local containers left stopped by the killed rsync session.
|
||||
# Remote containers deferred to docker_watchdog.sh.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# rsync_stop.sh
|
||||
# Auto-detect orchestrator. Kill rsync-only or full-stop accordingly.
|
||||
#
|
||||
# rsync_stop.sh --full-stop
|
||||
# Kill orchestrator first, then kill rsync. Nothing continues after this.
|
||||
#
|
||||
# rsync_stop.sh --rsync-only
|
||||
# Skip container recovery. Used when called by other scripts that handle
|
||||
# recovery themselves.
|
||||
#
|
||||
# rsync_stop.sh --dry-run
|
||||
# Show what would be killed without killing anything.
|
||||
#
|
||||
# rsync_stop.sh --status
|
||||
# Show local and remote rsync PIDs, running orchestrators, and lock files.
|
||||
#
|
||||
# rsync_stop.sh --full-stop --dry-run
|
||||
# Preview full-stop sequence without making any changes.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
DOCKER_TIMEOUT=15
|
||||
SSH_TIMEOUT=15
|
||||
|
||||
# ── Parse special flags before parse_args ─────────────────────────────────────────────────────
|
||||
FULL_STOP=false
|
||||
RSYNC_ONLY_MODE=false
|
||||
FILTERED_ARGS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--full-stop) FULL_STOP=true ;;
|
||||
--rsync-only) RSYNC_ONLY_MODE=true ;;
|
||||
*) FILTERED_ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — pkill and docker require root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# Soft IP resolution — rsync_stop continues local-only if remote unreachable
|
||||
REMOTE_REACHABLE=false
|
||||
REMOTE_SERVER=$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")
|
||||
if [[ -z "$REMOTE_SERVER" ]]; then
|
||||
warn "$REMOTE_SERVER_NAME — cannot resolve Tailscale IP, remote operations will be skipped"
|
||||
elif timeout "$SSH_TIMEOUT" ping -c1 -W3 "$REMOTE_SERVER" &>/dev/null; then
|
||||
REMOTE_REACHABLE=true
|
||||
log "$REMOTE_SERVER_NAME reachable ✅"
|
||||
else
|
||||
warn "$REMOTE_SERVER_NAME unreachable — remote operations will be skipped"
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
[[ "$FULL_STOP" == true ]] && warn "FULL STOP mode — orchestrator + rsync will be killed"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY RSYNC STOP STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_NET Remote: $REMOTE_ID ($REMOTE_SERVER_NAME)"
|
||||
echo ""
|
||||
|
||||
LOCAL_PIDS=$(pgrep -x rsync 2>/dev/null | tr '\n' ' ')
|
||||
echo " $ICON_SYNC Local rsync PIDs: ${LOCAL_PIDS:-none}"
|
||||
|
||||
for lockfile in "$LOCK_DIR"/*.lock; do
|
||||
[[ -f "$lockfile" ]] || continue
|
||||
content=$(cat "$lockfile" 2>/dev/null)
|
||||
pid="${content%%:*}"
|
||||
name="${content##*:}"
|
||||
[[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null && \
|
||||
echo " $ICON_RUNNING Lock: $name (PID $pid)"
|
||||
done
|
||||
|
||||
if [[ "$REMOTE_REACHABLE" == true ]]; then
|
||||
REMOTE_PIDS=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
root@"$REMOTE_SERVER" "pgrep -x rsync || true" 2>/dev/null | tr '\n' ' ')
|
||||
echo " $ICON_SYNC Remote rsync PIDs: ${REMOTE_PIDS:-none}"
|
||||
else
|
||||
echo " $ICON_WARN Remote: unreachable"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── ORCHESTRATOR DETECTION ────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Scans lock files to find which running process has rsync as a descendant.
|
||||
# No hardcoded script names — detects any orchestrator automatically.
|
||||
|
||||
detect_rsync_parent() {
|
||||
local rsync_pids
|
||||
rsync_pids=$(pgrep -x rsync 2>/dev/null || true)
|
||||
[[ -z "$rsync_pids" ]] && echo "" && return
|
||||
|
||||
for lockfile in "$LOCK_DIR"/*.lock; do
|
||||
[[ -f "$lockfile" ]] || continue
|
||||
local content pid locked_name
|
||||
content=$(cat "$lockfile" 2>/dev/null)
|
||||
pid="${content%%:*}"
|
||||
locked_name="${content##*:}"
|
||||
[[ -z "$pid" ]] && continue
|
||||
! kill -0 "$pid" 2>/dev/null && continue
|
||||
[[ "$locked_name" == rsync_* ]] && continue
|
||||
|
||||
local all_descendants
|
||||
all_descendants=$(pgrep -P "$pid" 2>/dev/null || true)
|
||||
|
||||
while IFS= read -r rsync_pid; do
|
||||
[[ -z "$rsync_pid" ]] && continue
|
||||
local ppid
|
||||
ppid=$(awk '/^PPid:/{print $2}' /proc/"$rsync_pid"/status 2>/dev/null || echo "")
|
||||
if echo "$all_descendants" | grep -qw "$rsync_pid" 2>/dev/null || \
|
||||
[[ "$ppid" == "$pid" ]]; then
|
||||
echo "${locked_name}:${pid}"
|
||||
return
|
||||
fi
|
||||
done <<< "$rsync_pids"
|
||||
done
|
||||
echo ""
|
||||
}
|
||||
|
||||
detect_rsync_parent_remote() {
|
||||
[[ "$REMOTE_REACHABLE" != true ]] && echo "" && return
|
||||
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
root@"$REMOTE_SERVER" bash << 'REMOTE_SCRIPT' 2>/dev/null
|
||||
LOCK_DIR="/tmp/unraid_locks"
|
||||
rsync_pids=$(pgrep -x rsync 2>/dev/null || true)
|
||||
[[ -z "$rsync_pids" ]] && exit 0
|
||||
for lockfile in "$LOCK_DIR"/*.lock; do
|
||||
[[ -f "$lockfile" ]] || continue
|
||||
content=$(cat "$lockfile" 2>/dev/null)
|
||||
pid="${content%%:*}"
|
||||
locked_name="${content##*:}"
|
||||
[[ -z "$pid" ]] && continue
|
||||
! kill -0 "$pid" 2>/dev/null && continue
|
||||
[[ "$locked_name" == rsync_* ]] && continue
|
||||
all_descendants=$(pgrep -P "$pid" 2>/dev/null || true)
|
||||
while IFS= read -r rsync_pid; do
|
||||
[[ -z "$rsync_pid" ]] && continue
|
||||
ppid=$(awk '/^PPid:/{print $2}' /proc/"$rsync_pid"/status 2>/dev/null || echo "")
|
||||
if echo "$all_descendants" | grep -qw "$rsync_pid" 2>/dev/null || \
|
||||
[[ "$ppid" == "$pid" ]]; then
|
||||
echo "${locked_name}:${pid}"
|
||||
exit 0
|
||||
fi
|
||||
done <<< "$rsync_pids"
|
||||
done
|
||||
REMOTE_SCRIPT
|
||||
}
|
||||
|
||||
LOCAL_ORCH=$(detect_rsync_parent)
|
||||
REMOTE_ORCH=""
|
||||
[[ "$REMOTE_REACHABLE" == true ]] && REMOTE_ORCH=$(detect_rsync_parent_remote)
|
||||
|
||||
# Determine mode
|
||||
if [[ "$FULL_STOP" == true ]]; then
|
||||
MODE="full-stop"
|
||||
elif [[ -n "$LOCAL_ORCH" ]] || [[ -n "$REMOTE_ORCH" ]]; then
|
||||
MODE="rsync-only"
|
||||
[[ -n "$LOCAL_ORCH" ]] && \
|
||||
warn "Local orchestrator detected: ${LOCAL_ORCH%%:*} — rsync-only mode"
|
||||
[[ -n "$REMOTE_ORCH" ]] && \
|
||||
warn "Remote orchestrator detected: ${REMOTE_ORCH%%:*} — rsync-only mode"
|
||||
warn "Use --full-stop to also kill the orchestrator"
|
||||
else
|
||||
MODE="rsync-only"
|
||||
log "No orchestrator detected — killing rsync directly"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Kill Orchestrators (full-stop only) ───────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
ORCHESTRATORS_KILLED=()
|
||||
REMOTE_ORCHESTRATORS_KILLED=()
|
||||
|
||||
kill_orchestrator() {
|
||||
local script_name="$1" pid="$2"
|
||||
local lockfile="$LOCK_DIR/${script_name}.lock"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would kill $script_name (PID $pid)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
kill -TERM "$pid" 2>/dev/null
|
||||
sleep 2
|
||||
kill -0 "$pid" 2>/dev/null && kill -KILL "$pid" 2>/dev/null
|
||||
sleep 1
|
||||
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
warn "$script_name stopped (PID $pid) ✅"
|
||||
rm -f "$lockfile"
|
||||
return 0
|
||||
else
|
||||
error "Failed to kill $script_name (PID $pid)"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ "$MODE" == "full-stop" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Kill Orchestrators ━━━"
|
||||
|
||||
if [[ -n "$LOCAL_ORCH" ]]; then
|
||||
local_name="${LOCAL_ORCH%%:*}"
|
||||
local_pid="${LOCAL_ORCH##*:}"
|
||||
warn "Killing local: $local_name (PID $local_pid)"
|
||||
kill_orchestrator "$local_name" "$local_pid" && \
|
||||
ORCHESTRATORS_KILLED+=("$local_name")
|
||||
else
|
||||
log "No local orchestrator running"
|
||||
fi
|
||||
|
||||
if [[ "$REMOTE_REACHABLE" == true ]] && [[ -n "$REMOTE_ORCH" ]]; then
|
||||
remote_name="${REMOTE_ORCH%%:*}"
|
||||
remote_pid="${REMOTE_ORCH##*:}"
|
||||
remote_lock="$LOCK_DIR/${remote_name}.lock"
|
||||
warn "Killing remote: $remote_name (PID $remote_pid)"
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
root@"$REMOTE_SERVER" \
|
||||
"kill -TERM '$remote_pid' 2>/dev/null; sleep 2; \
|
||||
kill -0 '$remote_pid' 2>/dev/null && kill -KILL '$remote_pid' 2>/dev/null; \
|
||||
rm -f '$remote_lock'" 2>/dev/null
|
||||
warn "Remote $remote_name stopped ✅"
|
||||
REMOTE_ORCHESTRATORS_KILLED+=("$remote_name")
|
||||
else
|
||||
warn "DRY RUN — would kill remote $remote_name (PID $remote_pid)"
|
||||
fi
|
||||
elif [[ "$REMOTE_REACHABLE" == true ]]; then
|
||||
log "No remote orchestrator running"
|
||||
fi
|
||||
|
||||
[[ ${#ORCHESTRATORS_KILLED[@]} -gt 0 || \
|
||||
${#REMOTE_ORCHESTRATORS_KILLED[@]} -gt 0 ]] && sleep 3
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Local Rsync ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Local Rsync ━━━"
|
||||
|
||||
LOCAL_KILLED=false
|
||||
LOCAL_PIDS=$(pgrep -x rsync 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$LOCAL_PIDS" ]]; then
|
||||
log "No rsync processes running locally"
|
||||
else
|
||||
warn "Found local rsync PIDs: $(echo "$LOCAL_PIDS" | tr '\n' ' ')"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would kill local rsync"
|
||||
else
|
||||
pkill -x rsync 2>/dev/null && LOCAL_KILLED=true || \
|
||||
warn "pkill returned non-zero — rsync may have already exited"
|
||||
[[ "$LOCAL_KILLED" == true ]] && warn "Local rsync killed ✅"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Clean stale rsync lock files
|
||||
for lockfile in "$LOCK_DIR"/rsync_*.lock; do
|
||||
[[ -f "$lockfile" ]] || continue
|
||||
content=$(cat "$lockfile" 2>/dev/null)
|
||||
pid="${content%%:*}"
|
||||
if [[ -n "$pid" ]] && ! kill -0 "$pid" 2>/dev/null; then
|
||||
log "Cleaning stale lock: $(basename "$lockfile")"
|
||||
[[ "$DRY_RUN" == false ]] && rm -f "$lockfile"
|
||||
fi
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Remote Rsync ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Remote Rsync — $REMOTE_SERVER_NAME ━━━"
|
||||
|
||||
REMOTE_KILLED=false
|
||||
|
||||
if [[ "$REMOTE_REACHABLE" == false ]]; then
|
||||
warn "Skipping — $REMOTE_SERVER_NAME unreachable"
|
||||
else
|
||||
REMOTE_PIDS=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
root@"$REMOTE_SERVER" "pgrep -x rsync || true" 2>/dev/null || true)
|
||||
|
||||
if [[ -z "$REMOTE_PIDS" ]]; then
|
||||
log "No rsync running on $REMOTE_SERVER_NAME"
|
||||
else
|
||||
warn "Found remote rsync PIDs: $(echo "$REMOTE_PIDS" | tr '\n' ' ')"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would kill remote rsync"
|
||||
else
|
||||
timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout="$SSH_TIMEOUT" \
|
||||
root@"$REMOTE_SERVER" "pkill -x rsync || true" 2>/dev/null && \
|
||||
REMOTE_KILLED=true || \
|
||||
warn "Remote pkill returned non-zero — rsync may have already exited"
|
||||
[[ "$REMOTE_KILLED" == true ]] && warn "Remote rsync killed ✅"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Recovery ━━━
|
||||
# ==============================================================================================
|
||||
# Restart local containers left stopped by interrupted rsync.
|
||||
# Remote containers left for docker_watchdog.sh to recover.
|
||||
# Skipped with --rsync-only flag (called by other scripts that handle recovery themselves).
|
||||
CONTAINERS_RESTARTED=()
|
||||
CONTAINERS_FAILED=()
|
||||
|
||||
if [[ "$RSYNC_ONLY_MODE" == false ]] && \
|
||||
{ [[ "$LOCAL_KILLED" == true ]] || [[ ${#ORCHESTRATORS_KILLED[@]} -gt 0 ]]; }; then
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_START Container Recovery ━━━"
|
||||
log "Checking profile containers for recovery..."
|
||||
|
||||
declare -A SEEN
|
||||
ALL_CONTAINERS=()
|
||||
|
||||
for profile_containers in "${PROFILE_CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
read -r -a container_list <<< "$profile_containers"
|
||||
for c in "${container_list[@]:-}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
if [[ -z "${SEEN[$c]:-}" ]]; then
|
||||
SEEN[$c]=1
|
||||
ALL_CONTAINERS+=("$c")
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
if [[ ${#ALL_CONTAINERS[@]} -eq 0 ]]; then
|
||||
log "No profile containers defined — skipping recovery"
|
||||
else
|
||||
for c in "${ALL_CONTAINERS[@]}"; do
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
|
||||
'{{.State.Running}}' "$c" 2>/dev/null || echo "unknown")
|
||||
case "$STATUS" in
|
||||
true)
|
||||
log "$c — running ✅"
|
||||
;;
|
||||
false)
|
||||
warn "$c — stopped — restarting..."
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $c"
|
||||
else
|
||||
if timeout "$DOCKER_TIMEOUT" docker start "$c" >/dev/null 2>&1; then
|
||||
warn "$c restarted ✅"
|
||||
CONTAINERS_RESTARTED+=("$c")
|
||||
else
|
||||
error "Failed to restart $c"
|
||||
CONTAINERS_FAILED+=("$c")
|
||||
fi
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
log "$c not found locally — skipping"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY RSYNC STOP SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Mode: $MODE"
|
||||
echo ""
|
||||
|
||||
echo "$ICON_HOST Local ($MY_ID):"
|
||||
[[ ${#ORCHESTRATORS_KILLED[@]} -gt 0 ]] && \
|
||||
warn " Orchestrators killed: ${ORCHESTRATORS_KILLED[*]}"
|
||||
if [[ "$LOCAL_KILLED" == true ]]; then
|
||||
warn " Rsync killed ✅"
|
||||
else
|
||||
log " No rsync was running"
|
||||
fi
|
||||
|
||||
echo "$ICON_NET Remote ($REMOTE_ID — $REMOTE_SERVER_NAME):"
|
||||
if [[ "$REMOTE_REACHABLE" == false ]]; then
|
||||
warn " Unreachable — skipped"
|
||||
else
|
||||
[[ ${#REMOTE_ORCHESTRATORS_KILLED[@]} -gt 0 ]] && \
|
||||
warn " Orchestrators killed: ${REMOTE_ORCHESTRATORS_KILLED[*]}"
|
||||
if [[ "$REMOTE_KILLED" == true ]]; then
|
||||
warn " Rsync killed ✅"
|
||||
else
|
||||
log " No rsync was running"
|
||||
fi
|
||||
fi
|
||||
|
||||
[[ ${#CONTAINERS_RESTARTED[@]} -gt 0 ]] && \
|
||||
warn "$ICON_CONTAINERS Containers recovered: ${CONTAINERS_RESTARTED[*]}"
|
||||
[[ ${#CONTAINERS_FAILED[@]} -gt 0 ]] && \
|
||||
echo "$ICON_ERROR Containers failed to restart: ${CONTAINERS_FAILED[*]}"
|
||||
|
||||
echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
else
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Notify if anything was actually killed or failed
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if [[ ${#CONTAINERS_FAILED[@]} -gt 0 ]]; then
|
||||
notify "Rsync stop on $(hostname) ($MY_ID) — containers failed to restart: ${CONTAINERS_FAILED[*]}" \
|
||||
"Rsync Stop" "warning"
|
||||
elif [[ "$LOCAL_KILLED" == true || "$REMOTE_KILLED" == true || \
|
||||
${#ORCHESTRATORS_KILLED[@]} -gt 0 ]]; then
|
||||
notify "Rsync stopped on $(hostname) ($MY_ID) — mode: $MODE${CONTAINERS_RESTARTED:+ — recovered: ${CONTAINERS_RESTARTED[*]}}" \
|
||||
"Rsync Stop" "warning"
|
||||
fi
|
||||
fi
|
||||
Executable
+339
@@ -0,0 +1,339 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Server Reboot ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Gracefully reboots the unRAID server with pre-flight checks, user warnings,
|
||||
# clean service shutdown, and disk sync. Use instead of raw /sbin/reboot —
|
||||
# gives users warning time and ensures services stop cleanly before the kernel
|
||||
# drops.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Shutdown Sequence
|
||||
# 1. Pre-flight warnings — rsync, mover, active Emby sessions (warn, not block)
|
||||
# 2. Wall message to all logged-in terminal users
|
||||
# 3. unRAID dashboard notification
|
||||
# 4. Wait REBOOT_SLEEP seconds — users time to save work
|
||||
# 5. array_stopping.sh — user scripts, rsync, mover, containers (verified stop)
|
||||
# 6. Graceful VM shutdown via virsh — ACPI signal, then wait REBOOT_VM_WAIT
|
||||
# 7. Stop libvirt (VM Manager)
|
||||
# 8. sync — filesystem buffers flushed to disk
|
||||
# 9. /sbin/reboot
|
||||
#
|
||||
# Pre-flight Warnings (informational — do not block)
|
||||
# rsync running → partial files possible if mid-transfer
|
||||
# mover running → files may be left mid-move on cache or array
|
||||
# Emby sessions → active streams/transcodes interrupted
|
||||
# Warnings do not block the reboot — you called this script, you know.
|
||||
#
|
||||
# VM Graceful Shutdown
|
||||
# virsh shutdown sends the ACPI power button signal — same as pressing the
|
||||
# physical power button. VM gets a chance to flush buffers and shut down.
|
||||
# After REBOOT_VM_WAIT seconds, libvirt stops anyway — reboot takes priority.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# /sbin/reboot requires root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent reboot calls.
|
||||
#
|
||||
# Host Identity in All Messages
|
||||
# detect_hosts() sets MY_ID — wall and notifications show which server is
|
||||
# rebooting. Critical on a two-server setup.
|
||||
#
|
||||
# sync Before Reboot
|
||||
# filesystem buffers flushed to disk before reboot command.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# REBOOT_SLEEP
|
||||
# Seconds between warning and shutdown sequence start. (default: 30)
|
||||
#
|
||||
# REBOOT_VM_WAIT
|
||||
# Seconds to wait for VMs to shut down gracefully. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# server_reboot.sh
|
||||
# Run pre-flight, warn users, stop services cleanly, then reboot.
|
||||
#
|
||||
# server_reboot.sh --dry-run
|
||||
# Walk through the entire shutdown sequence without stopping anything or rebooting.
|
||||
#
|
||||
# server_reboot.sh --status
|
||||
# Show running processes that would be affected: rsync, mover, VMs, containers.
|
||||
#
|
||||
# server_reboot.sh --reason="maintenance"
|
||||
# Include reason in wall message and notification. Defaults to "manual".
|
||||
#
|
||||
# server_reboot.sh --log
|
||||
# Verbose output — show each step of the shutdown sequence.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# ── Parse --reason flag before parse_args ─────────────────────────────────────────────────────
|
||||
REBOOT_REASON="manual"
|
||||
FILTERED_ARGS=()
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--reason=*) REBOOT_REASON="${arg#--reason=}" ;;
|
||||
*) FILTERED_ARGS+=("$arg") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — reboot requires root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
validate_int REBOOT_SLEEP "$REBOOT_SLEEP"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made, no reboot will occur"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY REBOOT STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_REBOOT Delay: ${REBOOT_SLEEP}s"
|
||||
echo "$ICON_GEAR VM wait: ${REBOOT_VM_WAIT:-30}s"
|
||||
echo "$ICON_GEAR Reason: $REBOOT_REASON"
|
||||
echo ""
|
||||
echo "━━━ Active Processes ━━━"
|
||||
|
||||
pgrep -x rsync >/dev/null 2>&1 && \
|
||||
warn " rsync: RUNNING — partial files if rebooted now" || \
|
||||
log " rsync: not running"
|
||||
|
||||
platform_is_mover_running && \
|
||||
warn " mover: RUNNING — files may be left mid-move" || \
|
||||
log " mover: not running"
|
||||
|
||||
if command -v virsh >/dev/null 2>&1; then
|
||||
VM_COUNT=$(virsh list --name 2>/dev/null | grep -c "." || echo 0)
|
||||
[[ "$VM_COUNT" -gt 0 ]] && \
|
||||
warn " VMs: $VM_COUNT running — will be gracefully shut down" || \
|
||||
log " VMs: none running"
|
||||
fi
|
||||
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
CONTAINER_COUNT=$(docker ps -q 2>/dev/null | wc -l || echo 0)
|
||||
log " Docker: $CONTAINER_COUNT container(s) running"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pre-flight Warnings ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SHIELD Pre-flight ━━━"
|
||||
WARNINGS=()
|
||||
|
||||
# rsync check — partial files if killed mid-transfer
|
||||
if pgrep -x rsync >/dev/null 2>&1; then
|
||||
RSYNC_PIDS=$(pgrep -x rsync | tr '\n' ' ')
|
||||
warn "rsync is running (PIDs: $RSYNC_PIDS) — partial files possible"
|
||||
warn "Consider: rsync_stop.sh before rebooting"
|
||||
WARNINGS+=("rsync running")
|
||||
fi
|
||||
|
||||
# mover check — files may be left mid-move
|
||||
if platform_is_mover_running; then
|
||||
warn "Mover is running — files may be left mid-move on cache or array"
|
||||
warn "Consider: mover_stop.sh before rebooting"
|
||||
WARNINGS+=("mover running")
|
||||
fi
|
||||
|
||||
# Emby sessions check — active streams interrupted
|
||||
if [[ -n "${EMBY_URL:-}" ]] && [[ -n "${EMBY_API_KEY:-}" ]]; then
|
||||
ACTIVE_STREAMS=$(curl -sf --max-time 5 \
|
||||
-H "X-Emby-Token: $EMBY_API_KEY" \
|
||||
"${EMBY_URL}/Sessions" 2>/dev/null | \
|
||||
grep -c "NowPlayingItem" 2>/dev/null || echo 0)
|
||||
ACTIVE_STREAMS="${ACTIVE_STREAMS//[^0-9]/}"
|
||||
if [[ "${ACTIVE_STREAMS:-0}" -gt 0 ]]; then
|
||||
warn "$ACTIVE_STREAMS active Emby stream(s) — will be interrupted"
|
||||
WARNINGS+=("${ACTIVE_STREAMS} Emby sessions")
|
||||
fi
|
||||
fi
|
||||
|
||||
CONTAINER_COUNT=$(docker ps -q 2>/dev/null | wc -l || echo 0)
|
||||
log "$ICON_CONTAINERS Docker: ${CONTAINER_COUNT} container(s) running"
|
||||
|
||||
if is_vm_manager_enabled && command -v virsh >/dev/null 2>&1; then
|
||||
VM_COUNT=$(virsh list --name 2>/dev/null | grep -c "." || echo 0)
|
||||
log "$ICON_GEAR VMs: ${VM_COUNT} running"
|
||||
fi
|
||||
|
||||
if [[ ${#WARNINGS[@]} -eq 0 ]]; then
|
||||
log "Pre-flight clean — no active processes to warn about"
|
||||
else
|
||||
warn "Proceeding with reboot despite warnings — ${WARNINGS[*]}"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Notify and Wait ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_REBOOT Reboot Sequence — $MY_ID ━━━"
|
||||
echo " Reason: $REBOOT_REASON"
|
||||
echo " Delay: ${REBOOT_SLEEP}s"
|
||||
echo " Dry Run: $DRY_RUN"
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
|
||||
if [[ "$REBOOT_SLEEP" -gt 0 ]]; then
|
||||
# Wall message — terminal users
|
||||
wall "$ICON_WARN $MY_ID ($LOCAL_SERVER_NAME) rebooting in ${REBOOT_SLEEP}s — reason: $REBOOT_REASON. Save your work now."
|
||||
|
||||
# unRAID notification — dashboard
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
notify "$MY_ID ($LOCAL_SERVER_NAME) rebooting in ${REBOOT_SLEEP}s — reason: $REBOOT_REASON${WARNINGS:+ — warnings: ${WARNINGS[*]}}" \
|
||||
"Server Reboot" "warning"
|
||||
fi
|
||||
|
||||
warn "Waiting ${REBOOT_SLEEP}s before shutdown sequence..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
sleep "$REBOOT_SLEEP"
|
||||
else
|
||||
warn "DRY RUN — skipping sleep"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Array Stop Orchestrator ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Array Stop Orchestrator ━━━"
|
||||
ARRAY_STOP_SCRIPT="$SCRIPT_DIR/../Orchestrators/array_stopping.sh"
|
||||
|
||||
if [[ ! -f "$ARRAY_STOP_SCRIPT" ]]; then
|
||||
warn "array_stopping.sh not found — skipping orchestrated stop"
|
||||
elif [[ "$DRY_RUN" == true ]]; then
|
||||
bash "$ARRAY_STOP_SCRIPT" --dry-run
|
||||
else
|
||||
if bash "$ARRAY_STOP_SCRIPT"; then
|
||||
log "Array stop complete ✅"
|
||||
else
|
||||
warn "array_stopping.sh reported failures — proceeding with reboot"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Graceful VM Shutdown ━━━
|
||||
# ==============================================================================================
|
||||
if is_vm_manager_enabled && command -v virsh >/dev/null 2>&1; then
|
||||
VM_LIST=$(virsh list --name 2>/dev/null | grep -v "^$" || true)
|
||||
if [[ -n "$VM_LIST" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Graceful VM Shutdown ━━━"
|
||||
while IFS= read -r vm; do
|
||||
[[ -z "$vm" ]] && continue
|
||||
warn "Sending ACPI shutdown to VM: $vm"
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
virsh shutdown "$vm" >/dev/null 2>&1 || true
|
||||
else
|
||||
warn "DRY RUN — would virsh shutdown $vm"
|
||||
fi
|
||||
done <<< "$VM_LIST"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
VM_WAIT="${REBOOT_VM_WAIT:-30}"
|
||||
log "Waiting ${VM_WAIT}s for VMs to shut down..."
|
||||
sleep "$VM_WAIT"
|
||||
fi
|
||||
else
|
||||
log "VM Manager enabled but no VMs running — skipping shutdown"
|
||||
fi
|
||||
else
|
||||
log "VM Manager not enabled — skipping VM shutdown"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Stop VM Manager ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Stop VM Manager ━━━"
|
||||
if ! is_vm_manager_enabled; then
|
||||
log "VM Manager not enabled — skipping"
|
||||
elif [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would stop VM Manager (libvirt)"
|
||||
else
|
||||
if platform_stop_service libvirt; then
|
||||
warn "VM Manager stopped ✅"
|
||||
else
|
||||
warn "VM Manager stop returned non-zero — may already be stopped"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Sync Disks ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_DISK Sync Disks ━━━"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would sync filesystem buffers"
|
||||
else
|
||||
sync
|
||||
log "Filesystem buffers flushed ✅"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Reboot ━━━
|
||||
# ==============================================================================================
|
||||
END=$(date +%s)
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY REBOOT SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_REBOOT Reason: $REBOOT_REASON"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
[[ ${#WARNINGS[@]} -gt 0 ]] && warn "Warnings: ${WARNINGS[*]}"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — sequence complete, no reboot executed"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
warn "$ICON_REBOOT Rebooting $MY_ID now..."
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
/sbin/reboot
|
||||
fi
|
||||
Executable
+153
@@ -0,0 +1,153 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Unraid API Key Renewal ====================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Creates/overwrites the Varaverk API key in the unraid-api service registry at
|
||||
# array start. The registry is ephemeral — OS updates and service restarts clear
|
||||
# it. This script re-registers the key every boot so Varaverk's enhanced
|
||||
# monitoring self-heals without manual intervention.
|
||||
#
|
||||
# Also updates HOST*_UNRAID_API_KEY in the local host conf so the partnership
|
||||
# page always reflects the live key value.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# unraid_api_key_renew.sh
|
||||
# Renew the key. Silent on success.
|
||||
#
|
||||
# unraid_api_key_renew.sh --dry-run
|
||||
# Show what would happen — no changes made.
|
||||
#
|
||||
# unraid_api_key_renew.sh --log
|
||||
# Verbose output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
acquire_lock
|
||||
detect_hosts
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
CONF_FILE="$SCRIPT_DIR/../Configurations/${MY_ID,,}.conf"
|
||||
VAR_NAME="${MY_ID}_UNRAID_API_KEY"
|
||||
|
||||
# Key name: "Varaverk <hostname>" stripping any unraid- prefix
|
||||
# Space separator — unRAID API only allows letters, numbers, and spaces
|
||||
HOSTNAME_SUFFIX=$(hostname -s 2>/dev/null | sed 's/^[Uu][Nn][Rr][Aa][Ii][Dd]-//' || hostname -s)
|
||||
KEY_NAME="Varaverk ${HOSTNAME_SUFFIX}"
|
||||
|
||||
log "$ICON_GEAR Conf file: $CONF_FILE"
|
||||
log "$ICON_GEAR Key var: $VAR_NAME"
|
||||
log "$ICON_GEAR Key name: $KEY_NAME"
|
||||
|
||||
if [[ ! -f "$CONF_FILE" ]]; then
|
||||
error "Conf file not found: $CONF_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would check registry for $KEY_NAME, renew only if missing"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
# Check if key already exists in the unraid-api registry before creating.
|
||||
# --overwrite generates a new key value every time, invalidating the old one.
|
||||
# Only renew if the registry has lost it.
|
||||
log "Checking unraid-api registry for $KEY_NAME..."
|
||||
EXISTING=$(timeout 5 /usr/local/sbin/unraid-api apikey --name "$KEY_NAME" --json </dev/null 2>/dev/null)
|
||||
KEY=$(echo "$EXISTING" | jq -r '.key // empty' 2>/dev/null)
|
||||
|
||||
if [[ -n "$KEY" ]]; then
|
||||
PREVIEW="${KEY:0:8}...${KEY: -4}"
|
||||
echo "API key valid ✅ — $VAR_NAME = $PREVIEW"
|
||||
log "Key found in registry — no renewal needed"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
log "Key not found in registry — creating new key..."
|
||||
|
||||
RAW=$(timeout 10 /usr/local/sbin/unraid-api apikey \
|
||||
--name "$KEY_NAME" --create --overwrite \
|
||||
--description "Varaverk plugin" --roles ADMIN --json </dev/null 2>&1)
|
||||
|
||||
if [[ -z "$RAW" ]]; then
|
||||
error "unraid-api returned no output"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
KEY=$(echo "$RAW" | jq -r '.key // empty' 2>/dev/null)
|
||||
if [[ -z "$KEY" ]]; then
|
||||
error "No key in unraid-api response: ${RAW:0:200}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────
|
||||
if grep -q "^\s*${VAR_NAME}\s*=" "$CONF_FILE"; then
|
||||
sed -i "s|^\(\s*${VAR_NAME}\s*=\s*\)\"[^\"]*\"|\1\"${KEY}\"|" "$CONF_FILE"
|
||||
else
|
||||
echo " ${VAR_NAME}=\"${KEY}\"" >> "$CONF_FILE"
|
||||
fi
|
||||
|
||||
PREVIEW="${KEY:0:8}...${KEY: -4}"
|
||||
log "Writing new key to: $CONF_FILE"
|
||||
warn "API key renewed ✅ — $VAR_NAME = $PREVIEW (registry had lost it)"
|
||||
|
||||
# ── Push renewed key into each partner's OWN conf ─────────────────────────────
|
||||
# Each host's conf is its complete keychest — no cross-host conf files needed.
|
||||
# SSH_KEY is set by detect_hosts() — this server's outbound private key.
|
||||
if [[ -z "$SSH_KEY" ]]; then
|
||||
log "No SSH key configured — skipping partner push"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
for host_var in $(compgen -v | grep -E '^HOST[0-9]+$'); do
|
||||
partner_host="${!host_var}"
|
||||
[[ -z "$partner_host" ]] && continue
|
||||
[[ "${host_var,,}" == "${MY_ID,,}" ]] && continue
|
||||
|
||||
partner_slot="${host_var,,}" # e.g. host2
|
||||
partner_ip=$(resolve_tailscale_ip "$partner_host" 2>/dev/null || true)
|
||||
[[ -z "$partner_ip" ]] && { log "Cannot resolve IP for $partner_host — skipping"; continue; }
|
||||
|
||||
# Target is the partner's OWN conf on their machine
|
||||
partner_conf="/boot/config/plugins/varaverk/Configurations/${partner_slot}.conf"
|
||||
tmp=$(mktemp /tmp/vv_kp_XXXXXX.sh)
|
||||
remote="/tmp/vv_kp_${RANDOM}.sh"
|
||||
chmod 700 "$tmp"
|
||||
|
||||
# Key stays in the temp file — never appears in SSH command args
|
||||
cat > "$tmp" <<PUSHSCRIPT
|
||||
#!/bin/sh
|
||||
target='${partner_conf}'
|
||||
if grep -q "\b${VAR_NAME}\b" "\$target" 2>/dev/null; then
|
||||
sed -i 's|^\(\\s*${VAR_NAME}\\s*=\\s*\)"[^"]*"|\1"${KEY}"|' "\$target"
|
||||
else
|
||||
printf ' ${VAR_NAME}="%s"\n' '${KEY}' >> "\$target"
|
||||
fi
|
||||
echo ok
|
||||
PUSHSCRIPT
|
||||
|
||||
if timeout 10 scp -i "$SSH_KEY" -o ConnectTimeout=10 -o BatchMode=yes \
|
||||
-o StrictHostKeyChecking=no "$tmp" "root@${partner_ip}:${remote}" 2>/dev/null; then
|
||||
if timeout 10 ssh -i "$SSH_KEY" -o ConnectTimeout=10 -o BatchMode=yes \
|
||||
-o StrictHostKeyChecking=no "root@${partner_ip}" \
|
||||
"bash '${remote}'; rc=\$?; rm -f '${remote}'; exit \$rc" 2>/dev/null | grep -q ok; then
|
||||
log "Key pushed to $partner_host ✅"
|
||||
else
|
||||
warn "Key push to $partner_host failed — they can create their own copy"
|
||||
fi
|
||||
else
|
||||
warn "SCP to $partner_host failed — skipping"
|
||||
fi
|
||||
rm -f "$tmp"
|
||||
done
|
||||
Executable
+256
@@ -0,0 +1,256 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= User Scripts Stop ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Stops all running User Script processes spawned by the unRAID User Scripts
|
||||
# plugin. Shows script names not just PIDs so you know what's being stopped.
|
||||
# Called automatically by server_reboot.sh as part of the shutdown sequence,
|
||||
# and useful directly when a script is stuck and won't respond to the UI.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Process Identification
|
||||
# Scans /proc/*/cmdline for processes whose command line contains
|
||||
# "/tmp/user.scripts". The User Scripts plugin stages all scripts in
|
||||
# /tmp/user.scripts/ before execution — more reliable than process name
|
||||
# matching which can vary.
|
||||
#
|
||||
# Stop Sequence Per Process
|
||||
# 1. Send SIGTERM — allows the script to trap and clean up gracefully
|
||||
# 2. Wait 5 seconds
|
||||
# 3. If still running → SIGKILL (force)
|
||||
# 4. Verify dead after SIGKILL — error if still running
|
||||
#
|
||||
# Self-Exclusion
|
||||
# If this script is run via the User Scripts plugin it would find its own
|
||||
# PID in the scan. Self-exclusion by PID prevents killing its own process
|
||||
# tree mid-execution.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# kill requires root for other users' processes.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent stop attempts.
|
||||
#
|
||||
# SIGTERM → SIGKILL Sequence
|
||||
# Graceful first. Forced only if SIGTERM ignored after 5 seconds.
|
||||
#
|
||||
# Post-Kill Verify
|
||||
# Confirms each process is actually dead. Errors and notifies if unkillable.
|
||||
#
|
||||
# Silent When Clean
|
||||
# No processes running = log() only, no visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# user_scripts_stop.sh
|
||||
# Find and stop all User Script processes. Silent if none running.
|
||||
#
|
||||
# user_scripts_stop.sh --dry-run
|
||||
# Show which processes would be stopped, with names and runtimes. No kills.
|
||||
#
|
||||
# user_scripts_stop.sh --status
|
||||
# Show currently running User Script processes with names and elapsed time.
|
||||
#
|
||||
# user_scripts_stop.sh --log
|
||||
# Verbose output — show each process found, each signal sent, each result.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
MY_PID=$$
|
||||
MY_PPID=$PPID
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root — kill requires root for other users' processes"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no processes will be killed"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER FUNCTIONS ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# Get script name from PID — extracts meaningful name from /tmp/user.scripts path
|
||||
get_script_name() {
|
||||
local pid="$1"
|
||||
local cmdline
|
||||
cmdline=$(tr '\0' ' ' < /proc/"$pid"/cmdline 2>/dev/null || echo "")
|
||||
# Extract the script filename from the /tmp/user.scripts/... path
|
||||
echo "$cmdline" | grep -o '/tmp/user\.scripts[^ ]*' | \
|
||||
awk -F/ '{print $NF}' | head -1 || echo "pid-$pid"
|
||||
}
|
||||
|
||||
# Get all user script PIDs — excludes self and own parent process tree
|
||||
get_user_script_pids() {
|
||||
local -a pids=()
|
||||
while IFS= read -r pid; do
|
||||
[[ -z "$pid" ]] && continue
|
||||
# Self-exclusion — don't kill our own process or parent
|
||||
[[ "$pid" == "$MY_PID" ]] && continue
|
||||
[[ "$pid" == "$MY_PPID" ]] && continue
|
||||
pids+=("$pid")
|
||||
done < <(
|
||||
for dir in /proc/[0-9]*/cmdline; do
|
||||
pid="${dir%/cmdline}"
|
||||
pid="${pid#/proc/}"
|
||||
if grep -ql '/tmp/user\.scripts' "$dir" 2>/dev/null; then
|
||||
echo "$pid"
|
||||
fi
|
||||
done
|
||||
)
|
||||
(( ${#pids[@]} > 0 )) && printf '%s\n' "${pids[@]}"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY USER SCRIPTS STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
|
||||
mapfile -t PIDS < <(get_user_script_pids)
|
||||
|
||||
if [[ ${#PIDS[@]} -eq 0 ]]; then
|
||||
log "No User Script processes running"
|
||||
else
|
||||
echo " ${#PIDS[@]} User Script process(es) running:"
|
||||
for pid in "${PIDS[@]}"; do
|
||||
name=$(get_script_name "$pid")
|
||||
elapsed=$(ps -o etimes= -p "$pid" 2>/dev/null | tr -d ' ')
|
||||
runtime=$(format_duration "${elapsed:-0}")
|
||||
echo " $ICON_RUNNING PID $pid — $name (${runtime})"
|
||||
done
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ User Scripts Stop ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_PLUGIN User Scripts Stop — $MY_ID ━━━"
|
||||
|
||||
START=$(date +%s)
|
||||
|
||||
mapfile -t PIDS < <(get_user_script_pids)
|
||||
|
||||
KILLED=()
|
||||
FAILED=()
|
||||
SKIPPED=()
|
||||
|
||||
if [[ ${#PIDS[@]} -eq 0 ]]; then
|
||||
echo "No User Script processes running — nothing to do"
|
||||
else
|
||||
warn "${#PIDS[@]} User Script process(es) found"
|
||||
echo ""
|
||||
|
||||
for pid in "${PIDS[@]}"; do
|
||||
name=$(get_script_name "$pid")
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would stop: $name (PID $pid)"
|
||||
SKIPPED+=("$name")
|
||||
continue
|
||||
fi
|
||||
|
||||
# Verify still running before trying to kill
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
log "$name (PID $pid) — already exited"
|
||||
continue
|
||||
fi
|
||||
|
||||
# SIGTERM — graceful stop
|
||||
log "Sending SIGTERM to $name (PID $pid)..."
|
||||
kill -TERM "$pid" 2>/dev/null || true
|
||||
sleep 5
|
||||
|
||||
# Check if stopped after SIGTERM
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
warn "Stopped: $name (PID $pid) ✅"
|
||||
KILLED+=("$name")
|
||||
continue
|
||||
fi
|
||||
|
||||
# SIGKILL — forced stop
|
||||
warn "$name still running after SIGTERM — sending SIGKILL"
|
||||
kill -KILL "$pid" 2>/dev/null || true
|
||||
sleep 2
|
||||
|
||||
# Final verify
|
||||
if ! kill -0 "$pid" 2>/dev/null; then
|
||||
warn "Force-stopped: $name (PID $pid) ✅"
|
||||
KILLED+=("$name")
|
||||
else
|
||||
error "Failed to kill: $name (PID $pid)"
|
||||
FAILED+=("$name")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY USER SCRIPTS STOP SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo ""
|
||||
|
||||
if [[ ${#PIDS[@]} -eq 0 ]]; then
|
||||
echo "No processes were running"
|
||||
elif [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would have stopped ${#SKIPPED[@]} process(es): ${SKIPPED[*]}"
|
||||
else
|
||||
[[ ${#KILLED[@]} -gt 0 ]] && warn "Stopped (${#KILLED[@]}): ${KILLED[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Failed (${#FAILED[@]}): ${FAILED[*]}"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ ${#FAILED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_ERROR Status: SOME PROCESSES COULD NOT BE KILLED"
|
||||
notify "User Scripts stop failed on $(hostname) ($MY_ID) — unkillable: ${FAILED[*]}" \
|
||||
"User Scripts Stop" "warning"
|
||||
else
|
||||
echo "$ICON_DONE Status: done ✅"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && exit 1
|
||||
exit 0
|
||||
Reference in New Issue
Block a user