feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -1,807 +1,176 @@
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# 🖥️ UNRAID ESSENTIALS
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# ━━━━━ UNRAID ESSENTIALS ━━━━━
|
||||
|
||||
**System-level scripts that act on the unRAID server itself — not containers, not
|
||||
media, not monitoring.** Keeping the server stable under load, recovering a frozen
|
||||
WebGUI, tuning kernel limits, suppressing log noise, and handling graceful shutdowns
|
||||
and reboots with proper warning sequences.
|
||||
**System-level scripts that act on the unRAID server itself — not containers,
|
||||
not media, not monitoring.** Keeping the server stable under load, recovering a
|
||||
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
|
||||
shutdowns with proper warning sequences.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Server Getting Into Unstable States With No Recovery Path**
|
||||
A container has a memory leak. RAM drops to 2GB. The system starts swapping. Docker
|
||||
watchdog tries to restart the container — but Docker itself is barely responding.
|
||||
The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||||
write log files. The server is functionally dead but still technically running.
|
||||
|
||||
Fix: `system_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||||
threshold breaches. Last line of defense before a hard crash.
|
||||
|
||||
**WebGUI Freezing and Nobody Noticing**
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state, or PHP-FPM workers
|
||||
are saturated, or emhttp has frozen. From a user perspective: dashboard doesn't
|
||||
load, settings don't save, containers can't be started or stopped via the UI. No
|
||||
container-level alert fires because this isn't a container problem — it's a web
|
||||
server problem. By the time someone notices it may have been broken for hours.
|
||||
|
||||
Fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Silent when healthy.
|
||||
|
||||
**50+ Containers Starting and Filling Syslog With Veth Noise**
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
|
||||
network interface for each one. Each interface generates multiple syslog entries.
|
||||
In the first minute after array start, syslog is buried under 200–400 lines of
|
||||
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
|
||||
— a failed mount, a permission error, a service that didn't start — are invisible.
|
||||
|
||||
Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
|
||||
starts. Applied at array start. Idempotent — silent when already correct.
|
||||
|
||||
**WebGUI Queuing Requests Under Load Without Explanation**
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Nothing in the logs
|
||||
explains it. The cause: PHP-FPM's `pm.max_children` defaults to 4–8 workers. With
|
||||
multiple users, active plugins, and 50+ containers potentially hitting the WebGUI,
|
||||
those workers saturate immediately. New requests queue behind active ones.
|
||||
|
||||
Fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~2MB = ~500MB total. On 128GB this is trivially small.
|
||||
|
||||
**inotify Exhaustion Producing Unexplained Failures**
|
||||
When inotify limits are exhausted, containers silently stop receiving filesystem
|
||||
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
|
||||
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
|
||||
and all containers share the same pool.
|
||||
|
||||
Fix: `inotify_tuning.sh` — raises all three inotify limits at array start. Must
|
||||
run FIRST in ARRAY_START_SCRIPTS before any containers start.
|
||||
|
||||
**Mover Getting Killed Mid-Transfer Leaving Files Inconsistent**
|
||||
The mover is running — moving a large batch of files from cache to array. A reboot
|
||||
is triggered. The mover stops mid-file. The file exists partially on both cache and
|
||||
array simultaneously. unRAID's deduplication layer is confused.
|
||||
|
||||
Fix: `mover_stop.sh` — warns users via wall message, waits the configured timeout,
|
||||
SIGTERM (graceful — finishes current file), SIGKILL only if needed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
```
|
||||
unRAID_Essentials/ ← acts on the server itself (this folder)
|
||||
Docker_Essentials/ ← acts on containers
|
||||
Media/ ← acts on the library
|
||||
Monitors/ ← observes and reports
|
||||
```
|
||||
|
||||
> **The escalation chain matters here.** Docker_Essentials handles container-level
|
||||
> problems. unRAID_Essentials handles server-level problems. The watchdogs are
|
||||
> designed to work together — docker_watchdog.sh heals containers first,
|
||||
> system_watchdog.sh reboots only when healing has failed.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Server Getting Into Unstable States With No Recovery Path
|
||||
|
||||
A container has a severe memory leak. RAM drops to 2GB. The system starts swapping.
|
||||
Everything slows down. Docker watchdog tries to restart the container — but Docker
|
||||
itself is barely responding. The restart hangs. The watchdog is stuck. Nothing is
|
||||
getting better. The server needs a reboot, but nothing in the ecosystem is authorised
|
||||
to call one.
|
||||
|
||||
Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The WebGUI
|
||||
shows nothing useful. The server is functionally dead but still technically running.
|
||||
Again — needs a reboot, nothing calls one.
|
||||
|
||||
The fix: `system_watchdog.sh` — the last line of defense. Three-tier response:
|
||||
immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, and a
|
||||
strike system for sustained threshold breaches. When everything else has failed,
|
||||
system_watchdog reboots cleanly before a hard crash happens.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 WebGUI Freezing and Nobody Noticing
|
||||
|
||||
The WebGUI becomes unresponsive. Nginx gets into a bad state. Or PHP-FPM workers are
|
||||
saturated and new requests are queueing indefinitely. Or emhttp itself has frozen.
|
||||
From a user perspective: dashboard doesn't load, settings don't save, containers
|
||||
can't be started or stopped via the UI.
|
||||
|
||||
Nothing in the container stack alerts on a frozen WebGUI — it's not a container
|
||||
problem, it's a web server problem. The only way to know is if someone tries to use
|
||||
the UI and notices. By which point it may have been broken for hours.
|
||||
|
||||
The fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx →
|
||||
php-fpm → emhttp. Lightest fix first. Notifies on any restart so you know it happened.
|
||||
Silent when healthy — 144 runs per day with no output is the correct behaviour.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 50+ Containers Starting and Filling Syslog With Veth Noise
|
||||
|
||||
Array starts. 50+ containers come up simultaneously. Docker creates a virtual network
|
||||
interface for each one. Each creation generates multiple syslog entries. In the first
|
||||
few minutes after array start the syslog is buried under hundreds of lines of:
|
||||
|
||||
```
|
||||
kernel: veth2a3b4c5: renamed from eth0
|
||||
kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||||
kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||||
```
|
||||
|
||||
Real events — a failed mount, a permission error, a service that didn't start —
|
||||
are invisible in this noise. And on a busy server that restarts containers regularly,
|
||||
this noise continues throughout the day.
|
||||
|
||||
The fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any
|
||||
container starts. Applied at array start, idempotent, silent when already correct.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 WebGUI Queueing Requests Under Load Without Explanation
|
||||
|
||||
The WebGUI feels slow. Clicking a button takes 5 seconds. Saving settings seems to
|
||||
hang. Nothing in the logs explains it. Container starts from the UI timeout. The
|
||||
server itself is not under load — CPU is fine, RAM is fine.
|
||||
|
||||
The cause: PHP-FPM's `pm.max_children` defaults to 4-8 workers. On a server with
|
||||
multiple users, active plugins, automated tools polling the API, and 50+ containers
|
||||
all potentially hitting the WebGUI simultaneously, those 4-8 workers saturate
|
||||
immediately. New requests queue behind active ones. Everything feels slow.
|
||||
|
||||
The fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start.
|
||||
250 workers × ~40MB = ~10GB worst case. On 128GB this is trivially small. The
|
||||
WebGUI becomes responsive immediately. Idempotent — silent when already correct.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 inotify Exhaustion Producing Unexplained Failures
|
||||
|
||||
Already documented in README-Monitors.md (system_tuning_monitor.sh section). Short
|
||||
version: when inotify limits are exhausted, containers silently stop receiving file
|
||||
system events. Downloads complete but arrs don't detect them. The kernel hits the
|
||||
limit and new watches fail silently. VSCode shows "unable to watch for file changes"
|
||||
and misses edits.
|
||||
|
||||
The fix: `inotify_tuning.sh` — raises all three inotify limits at array start.
|
||||
1M watches (raised from 512K — Code-Server with node_modules needs this), 1024
|
||||
instances, 32768 queued events. The startup race note: if Code-Server starts before
|
||||
this runs, it inherits old limits. Restart Code-Server if the VSCode error appears
|
||||
after limits are applied.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Mover Getting Killed Mid-Transfer Leaving Files Inconsistent
|
||||
|
||||
The mover is running — moving a large batch of files from cache to array. Someone
|
||||
clicks reboot from the UI. Or a script kills the mover process directly. The mover
|
||||
stops mid-file. The file exists partially on both cache and array simultaneously.
|
||||
unRAID's deduplication layer is confused. The file is inaccessible.
|
||||
|
||||
The fix: `mover_stop.sh` — warns logged-in users via wall message, waits the
|
||||
configured timeout, then sends SIGTERM (graceful) and verifies. The mover gets to
|
||||
finish its current file operation before stopping. SIGKILL is a last resort with a
|
||||
warning that partial files may exist.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Four distinct server-level roles:
|
||||
|
||||
```
|
||||
🛡️ Last-resort stability system_watchdog.sh — reboots before crash
|
||||
🌐 WebGUI availability webgui_restart.sh — recovers frozen UI
|
||||
⚙️ Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
🔇 Log hygiene docker_syslog_filter.sh — suppress veth noise
|
||||
clear_logs.sh — weekly log trimming
|
||||
🔄 Graceful operations mover_stop.sh — clean mover stop
|
||||
server_reboot.sh — clean reboot with warning
|
||||
user_scripts_stop.sh — stop running scripts
|
||||
Last-resort stability system_watchdog.sh — reboots before crash
|
||||
Pressure reduction resource_watchdog.sh — throttle/pause/stop under load
|
||||
WebGUI availability webgui_restart.sh — nginx → php-fpm → emhttp escalation
|
||||
Kernel tuning inotify_tuning.sh — file watch limits
|
||||
php_fpm_max_children.sh — PHP worker count
|
||||
Log hygiene docker_syslog_filter.sh — suppress veth noise at start
|
||||
clear_logs.sh — weekly log trimming
|
||||
Graceful operations mover_stop.sh — clean mover stop
|
||||
rsync_stop.sh — smart rsync stop (orchestrator-aware)
|
||||
user_scripts_stop.sh — stop running User Scripts
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
| Script | Purpose | When |
|
||||
|--------|---------|------|
|
||||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop |
|
||||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | Every 10 minutes |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM worker count | At array start |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth log noise | At array start |
|
||||
| `clear_logs.sh` | Size-threshold weekly log cleanup | Weekly via maintenance window |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual or called by system_watchdog |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual or called by server_reboot |
|
||||
```
|
||||
Orchestrators/
|
||||
array_start.sh ─────────────────────────► inotify_tuning.sh (first in sequence)
|
||||
─────────────────────────► docker_syslog_filter.sh (second)
|
||||
─────────────────────────► php_fpm_max_children.sh
|
||||
─────────────────────────► system_watchdog.sh (background loop)
|
||||
|
||||
---
|
||||
watchdog_orchestrator.sh ───────────────► resource_watchdog.sh (every minute)
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🛡️ system_watchdog.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
|
||||
The last line of defense. Reboots the system cleanly before it crashes uncleanly.
|
||||
Three-tier response system — critical failures bypass everything and reboot immediately,
|
||||
OOM-confirmed crises bypass the strike system, sustained threshold breaches use strikes.
|
||||
Runs continuously as a background process started by `array_start.sh`.
|
||||
|
||||
> Full architecture documentation in `README-Docker_Essentials.md` under
|
||||
> "Relationship to System Watchdog". This section covers the system_watchdog itself.
|
||||
|
||||
```bash
|
||||
# Started by: array_start.sh (continuous background process)
|
||||
# Interval: SYSTEM_WATCHDOG_INTERVAL=300 (5 minutes)
|
||||
Docker_Essentials/
|
||||
docker_watchdog.sh ◄─── reads ──────────── resource_watchdog.sh state
|
||||
(mem_shutdown_active flag)
|
||||
```
|
||||
|
||||
`system_watchdog.sh` and `docker_watchdog.sh` (in Docker_Essentials/) are
|
||||
designed to work together — docker_watchdog heals containers first,
|
||||
system_watchdog reboots only when healing has failed. `resource_watchdog.sh`
|
||||
coordinates with docker_watchdog via the `mem_shutdown_active` state flag to
|
||||
prevent docker_watchdog from restarting containers that resource_watchdog just
|
||||
stopped to free RAM.
|
||||
|
||||
---
|
||||
|
||||
### ── Three-Tier Response System ──────────────────────────────────────────────
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# TIER 1 — CRITICAL (bypass ALL strikes, reboot immediately)
|
||||
# These failures are acute — the system is not recoverable by waiting.
|
||||
# Single detection = immediate reboot. No confirmation window.
|
||||
#
|
||||
# Docker daemon unresponsive:
|
||||
# Attempt /etc/rc.d/rc.docker restart first.
|
||||
# Wait 15 seconds. Verify daemon responding.
|
||||
# If still hung → CRITICAL reboot.
|
||||
# A hung daemon cannot be healed — every subsequent docker command hangs.
|
||||
#
|
||||
# rootfs at ROOTFS_CRITICAL_PCT (99%+):
|
||||
# Writes are failing. SSH may stop. Logs can't be written.
|
||||
# Nothing can be fixed from this state without a reboot.
|
||||
#
|
||||
# Kernel BUG/Oops in dmesg:
|
||||
# Kernel running with corrupted state.
|
||||
# Delta-based: new oops since last cycle → reboot.
|
||||
#
|
||||
# File descriptor exhaustion (FD_CRITICAL_PCT=95%):
|
||||
# New connections failing. Docker can't spawn processes. SSH drops.
|
||||
#
|
||||
# /boot read-only:
|
||||
# State files and config writes silently failing.
|
||||
# Write test on /boot every cycle.
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop via array_start.sh |
|
||||
| `resource_watchdog.sh` | Pressure reduction — throttle/pause/stop under load | Every minute via watchdog_orchestrator.sh |
|
||||
| `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp | Every 10 min via User Scripts |
|
||||
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — FIRST |
|
||||
| `php_fpm_max_children.sh` | Set PHP-FPM max worker count | At array start |
|
||||
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||||
| `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual / before reboot |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual |
|
||||
| `user_scripts_stop.sh` | Stop all running User Script processes | Manual / called by server_reboot.sh |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||
|
||||
# TIER 2 — URGENT (bypass strikes when OOM confirms crisis)
|
||||
# RAM < MEM_GB (4GB) AND OOM kills >= OOM_LIMIT (3) in this cycle
|
||||
# Without OOM confirmation → standard strike system applies.
|
||||
# Rationale: 1-2 OOM kills = docker_watchdog.sh handles it.
|
||||
# 3+ kills while RAM critical = system dying faster than watchdogs heal.
|
||||
# OOM victims from dmesg included in reboot message (diagnostic context).
|
||||
---
|
||||
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
# TIER 3 — STANDARD (strike system — N consecutive failures → reboot)
|
||||
# Everything else: RAM tiers, load, CPU temp, zombies, /var/log,
|
||||
# /tmp, containers, NIC, mdstat, sshd
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
Array starts
|
||||
│
|
||||
├─ inotify_tuning.sh ← FIRST — kernel limits inherited at container launch
|
||||
├─ docker_syslog_filter.sh ← SECOND — before any veth interfaces are created
|
||||
├─ php_fpm_max_children.sh ← before WebGUI is under load
|
||||
└─ system_watchdog.sh ← starts background loop
|
||||
|
||||
---
|
||||
|
||||
### ── RAM Tiers ────────────────────────────────────────────────────────────────
|
||||
Every minute (watchdog_orchestrator.sh):
|
||||
└─ resource_watchdog.sh
|
||||
Level 1 (soft): throttle SABnzbd + qBit download speeds
|
||||
Level 2 (medium): further throttle + docker pause non-essential containers
|
||||
Level 3 (hard): docker stop optional services + set mem_shutdown_active=true
|
||||
↓
|
||||
docker_watchdog.sh reads mem_shutdown_active — defers restarts
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Three-level graduated response — not a single threshold.
|
||||
#
|
||||
SYS_WATCHDOG_MEM_WARN_GB=10 # warn + notify once — informational only
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_GB=6 # stop non-essential containers, wait for recovery
|
||||
SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (or bypass if OOM confirms)
|
||||
SYS_WATCHDOG_MEM_RECOVER_GB=30 # RAM must reach this before restarting containers
|
||||
#
|
||||
# Containers excluded from RAM emergency shutdown — these stay running:
|
||||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||||
"NginxProxyManager" # external access — stop this and users lose everything
|
||||
"Authelia" # auth — without this nothing is accessible
|
||||
"Mariadb" # Authelia dependency
|
||||
"Redis" # Authelia dependency
|
||||
"Emby" # media server — Live TV buffering
|
||||
"Dispatcharr" # Live TV scheduler — loses state if stopped
|
||||
)
|
||||
#
|
||||
# Coordination with docker_watchdog.sh:
|
||||
# system_watchdog writes mem_shutdown_active=true to SYS_WATCHDOG_STATE_FILE.
|
||||
# docker_watchdog reads this flag and defers ALL container restart logic.
|
||||
# Without this: both watchdogs fight — system_watchdog stops containers,
|
||||
# docker_watchdog restarts them, RAM never recovers.
|
||||
# With this: docker_watchdog stands down until mem_shutdown_active clears.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
Every 10 minutes (User Scripts):
|
||||
└─ webgui_restart.sh
|
||||
WebGUI OK → silent exit
|
||||
Not responding:
|
||||
Step 1: restart nginx → recheck
|
||||
Step 2: restart php-fpm → recheck
|
||||
Step 3: restart emhttp → recheck
|
||||
All failed → notify, exit 1
|
||||
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
System logs: clear if > LOG_MIN_SIZE_MB
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
|
||||
Manual operations:
|
||||
mover_stop.sh → wall → SIGTERM → SIGKILL → verify stopped
|
||||
rsync_stop.sh → detect orchestrator → kill rsync (or orchestrator+rsync)
|
||||
user_scripts_stop.sh → scan /proc → SIGTERM → SIGKILL per process
|
||||
server_reboot.sh → pre-flight → wall → wait → VMs → Docker → sync → reboot
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Per-Host Check Toggles ───────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf (all checks in master_host*.conf — not master.conf)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Each check is independently toggleable per server.
|
||||
# HOST1 and HOST2 may have different hardware and different workloads.
|
||||
# detect_hosts() aliases HOST*_SYS_WATCHDOG_CHECK_* → SYS_WATCHDOG_CHECK_*
|
||||
#
|
||||
# Tier 1 — Critical (bypass strikes):
|
||||
HOST1_SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_ROOTFS=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_KERNEL_OOPS=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_FD=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_BOOT=true
|
||||
|
||||
# Tier 2 — Urgent (OOM bypass):
|
||||
HOST1_SYS_WATCHDOG_CHECK_OOM=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_RAM=true
|
||||
|
||||
# Tier 3 — Standard (strike system):
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOG=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_ARC=true # HOST1 runs ZFS — enable
|
||||
HOST1_SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOAD=false # disabled — transcoding causes normal spikes
|
||||
HOST1_SYS_WATCHDOG_CHECK_ZOMBIES=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_CONTAINERS=true # monitors docker_watchdog persistent skip list
|
||||
HOST1_SYS_WATCHDOG_CHECK_TMP=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_MDSTAT=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_NETWORK=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_SSHD=true
|
||||
HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false # disabled — may false positive during encoding
|
||||
HOST1_SYS_WATCHDOG_NIC="eth0" # verify: ip link show | grep "^[0-9]"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Abort Conditions ─────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Conditions that prevent a reboot even when a threshold is hit.
|
||||
# CRITICAL tier bypasses these — truly critical conditions reboot regardless.
|
||||
#
|
||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # rebooting with bad pool risks data loss
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity is better than crashing
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting mover is better than crashing
|
||||
#
|
||||
# Philosophy: a graceful reboot before a crash is always better than a hard crash.
|
||||
# The abort conditions protect against the cases where a reboot itself causes harm
|
||||
# (data loss from bad ZFS pool). Parity and mover can be restarted after reboot.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Reboot Loop Protection ───────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# If the server keeps rebooting, something is wrong that rebooting isn't fixing.
|
||||
# After REBOOT_LIMIT reboots in REBOOT_WINDOW_HRS → shutdown instead.
|
||||
# Shutdown prevents: hardware damage, filesystem corruption from repeated reboots,
|
||||
# infinite loop that never lets you investigate.
|
||||
# State file: /boot/config/system_watchdog_reboots.db — survives reboots.
|
||||
#
|
||||
SYS_WATCHDOG_REBOOT_LIMIT=3 # reboots before shutdown instead
|
||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12 # rolling window in hours
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── State File Heartbeat ─────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# system_watchdog writes watchdog_cycle=N to SYS_WATCHDOG_STATE_FILE every cycle.
|
||||
# This keeps the file's modification time current.
|
||||
#
|
||||
# docker_watchdog.sh uses the state file mtime as a stale guard — if the file
|
||||
# is more than 2 hours old while mem_shutdown_active=true is set, system_watchdog
|
||||
# may have stopped running. docker_watchdog resumes normal operation rather than
|
||||
# being silenced indefinitely by a stale flag.
|
||||
#
|
||||
# Without this heartbeat: if all standard checks pass and no state writes happen
|
||||
# (e.g. CHECK_KERNEL_OOPS=false AND CHECK_MDSTAT=false), the file mtime could go
|
||||
# stale even with the watchdog running.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # normal (continuous — started by array_start.sh)
|
||||
system_watchdog.sh --dry-run # trigger detection without rebooting
|
||||
system_watchdog.sh --status # show all tiers, thresholds, active strikes, RAM state
|
||||
system_watchdog.sh --log # verbose per-cycle output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🌐 webgui_restart.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Monitors the unRAID WebGUI availability and recovers it automatically when unresponsive.
|
||||
Three-step escalating strategy — lightest fix first, heaviest last. Silent when healthy.
|
||||
|
||||
```bash
|
||||
# Scheduled: */10 * * * * (every 10 minutes)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Three-Step Escalation ────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Step 1 — nginx restart
|
||||
# Lightest fix — handles most WebGUI failures.
|
||||
# nginx is the web server layer. Crash, worker stuck, connection timeout.
|
||||
# Wait WEBGUI_NGINX_WAIT seconds → curl recheck.
|
||||
#
|
||||
# Step 2 — php-fpm restart
|
||||
# Added because WebGUI can appear frozen due to PHP worker exhaustion.
|
||||
# pm.max_children workers all occupied → new requests queue → dashboard hangs.
|
||||
# php-fpm restart far less disruptive than emhttp.
|
||||
# Wait WEBGUI_PHP_WAIT seconds → curl recheck.
|
||||
# Note: system_tuning_monitor.sh tracks worker saturation over time.
|
||||
#
|
||||
# Step 3 — emhttp restart
|
||||
# Heaviest fix. emhttp is the core unRAID management daemon.
|
||||
# Array, Docker, shares continue running — only WebGUI management restarts.
|
||||
# Takes longer to recover — WEBGUI_EMHTTP_WAIT gives it time.
|
||||
# Wait WEBGUI_EMHTTP_WAIT seconds → curl recheck.
|
||||
#
|
||||
# If all three fail → notify warning — manual intervention needed.
|
||||
# Guidance: pgrep -x nginx emhttp | check journalctl | consider server_reboot.sh
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Configuration ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WEBGUI_URL="http://localhost" # adjust if non-standard port
|
||||
WEBGUI_TIMEOUT=5 # seconds before curl times out
|
||||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart — takes longer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
webgui_restart.sh # check once — silent if healthy, escalates if not
|
||||
webgui_restart.sh --dry-run # walk through escalation without restarting anything
|
||||
webgui_restart.sh --status # show WebGUI state + nginx/php-fpm/emhttp process state
|
||||
webgui_restart.sh --log # verbose — show each escalation step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 📡 inotify_tuning.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Raises Linux inotify kernel limits at array start. Idempotent — completely silent
|
||||
when values are already correct.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh — FIRST in ARRAY_START_SCRIPTS)
|
||||
# Must run before containers start — containers inherit limits at startup
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Three Limits ─────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
INOTIFY_MAX_INSTANCES=1024 # default: 128 — max inotify instances per user
|
||||
# 1024 handles ~20-30 containers watching files
|
||||
|
||||
INOTIFY_MAX_WATCHES=1048576 # default: 8192 — SHARED budget across ALL users/containers
|
||||
# Was 512K — raised to 1M (1048576)
|
||||
# VSCode/Code-Server alone needs ~50K-200K for large workspaces
|
||||
# with node_modules. All arr containers + Emby + VSCode share
|
||||
# this budget. 1M safe on 128GB RAM (~128MB kernel memory)
|
||||
# If VSCode shows "unable to watch for file changes" → too low
|
||||
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768 # default: 16384 — events buffered before dropping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Startup Race Condition ───────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# inotify limits are kernel-wide — they take effect immediately on sysctl write.
|
||||
# But containers that have ALREADY started inherit the OLD limits at startup.
|
||||
# Those containers keep their inherited (low) limits until restarted.
|
||||
#
|
||||
# This is why inotify_tuning.sh must be FIRST in ARRAY_START_SCRIPTS — before
|
||||
# any container starts. If Code-Server starts before limits are raised:
|
||||
# → Code-Server inherits old 8192 watch limit
|
||||
# → VSCode shows "unable to watch for file changes"
|
||||
# → Fix: docker restart Code-Server (picks up current kernel limits on start)
|
||||
#
|
||||
# The script warns if it changed any values:
|
||||
# "If Code-Server is running: docker restart Code-Server"
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
inotify_tuning.sh # normal run (idempotent — silent when correct)
|
||||
inotify_tuning.sh --dry-run # show what would change without changing
|
||||
inotify_tuning.sh --status # current values vs targets + top inotify consumers
|
||||
inotify_tuning.sh --log # verbose — show each sysctl write
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ⚙️ php_fpm_max_children.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Raises PHP-FPM `pm.max_children` at array start to prevent WebGUI queueing under
|
||||
load. Idempotent — completely silent when already correct. No PHP-FPM restart unless
|
||||
value actually changed.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Why 250 Workers ──────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
PHP_MAX_CHILDREN=250 # default: 4-8 — far too low for a busy server
|
||||
# Each worker: ~2MB resident memory at idle
|
||||
# 250 workers × 2MB = ~500MB — trivial on 128GB
|
||||
# Worst case (all active): ~250 × 40MB = ~10GB
|
||||
# In practice: rarely all active simultaneously
|
||||
# On 64GB (HOST2): still appropriate — 250 × 40MB
|
||||
# = 10GB worst case = 15% of RAM, acceptable
|
||||
#
|
||||
# Without this fix:
|
||||
# 5 users hit the WebGUI simultaneously → 8 workers exhausted → 5 more queue
|
||||
# Each queued request waits for a worker to free → 5-10 second response times
|
||||
# Looks like a slow server — it's just a queue
|
||||
#
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf" # path may change with PHP version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
php_fpm_max_children.sh # normal run (idempotent)
|
||||
php_fpm_max_children.sh --dry-run # show what would change
|
||||
php_fpm_max_children.sh --status # show current value vs target + worker count
|
||||
php_fpm_max_children.sh --log # verbose — show config write + restart
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔇 docker_syslog_filter.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Creates an rsyslog drop rule for Docker veth/docker0 interface noise. Idempotent —
|
||||
silent when filter already correct, only writes + restarts rsyslog when something changed.
|
||||
|
||||
```bash
|
||||
# Scheduled: At Startup of Array (via array_start.sh — before containers start)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What Gets Suppressed ─────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# These kernel messages are generated on every container start and stop:
|
||||
#
|
||||
# kernel: veth2a3b4c5: renamed from eth0
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered blocking state
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered forwarding state
|
||||
# kernel: docker0: port 1(veth2a3b4c5) entered disabled state
|
||||
#
|
||||
# 50+ containers at array start = 200-400 lines of this in the first minute.
|
||||
# Containers restart throughout the day = continuous noise.
|
||||
# Real events buried and invisible in syslog.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Idempotent Design ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Runs at every array start but only changes something when needed:
|
||||
# Filter file exists and content is correct → exit 0 silently
|
||||
# Filter file missing or content changed → write + restart rsyslog
|
||||
#
|
||||
# Expected content compared exactly — single source of truth:
|
||||
EXPECTED_FILTER='if ($msg contains "veth" or $msg contains "docker0") then {
|
||||
stop
|
||||
}'
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
docker_syslog_filter.sh # normal run (idempotent — silent when correct)
|
||||
docker_syslog_filter.sh --dry-run # show what would be written without writing
|
||||
docker_syslog_filter.sh --status # show current filter file + rsyslog state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🗑️ clear_logs.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Clears system logs and Docker container logs using size thresholds. Only clears logs
|
||||
large enough to be worth clearing — preserves recent diagnostic context on small logs.
|
||||
|
||||
```bash
|
||||
# Called by: weekly_sync_maintenance.sh via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Size Threshold Approach ──────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Why thresholds instead of clearing everything:
|
||||
# A 2MB syslog contains useful recent history — not worth clearing.
|
||||
# A 500MB Docker log is consuming rootfs and contains mostly noise — clear it.
|
||||
# Blind truncation destroys diagnostic context for no benefit.
|
||||
#
|
||||
LOG_MIN_SIZE_MB=10 # skip system log if under this size — keep history
|
||||
LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this size
|
||||
# Active containers (Emby, SABnzbd) grow fastest
|
||||
# 100MB × 30 containers = 3GB before any clearing kicks in
|
||||
|
||||
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
|
||||
#
|
||||
# Truncation not logrotate:
|
||||
# unRAID writes to tmpfs (/var/log) — logrotate's compress + archive approach
|
||||
# would consume more tmpfs space, not less.
|
||||
# : > file keeps the file descriptor valid while emptying content.
|
||||
# Safe for running services (syslogd continues writing to the same fd).
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
clear_logs.sh # normal run — silent if all logs under threshold
|
||||
clear_logs.sh --dry-run # show what would be cleared and sizes
|
||||
clear_logs.sh --status # show current log sizes vs thresholds
|
||||
clear_logs.sh --log # verbose — show each file evaluated
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ⏹️ mover_stop.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops the unRAID mover cleanly — wall warning, configurable timeout, SIGTERM → verify
|
||||
→ SIGKILL sequence. Safe to run when mover is not running — exits cleanly with a log.
|
||||
|
||||
---
|
||||
|
||||
### ── Stop Sequence ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Check if mover is running — exit cleanly if not
|
||||
# 2. Wall message: "HOST1 (unRAID-Gmer4Lfe) — mover stopping in 30s"
|
||||
# MY_ID included — on shared terminal it's clear which server
|
||||
# 3. Wait MOVER_STOP_TIMEOUT seconds
|
||||
# 4. SIGTERM — allows mover to finish its current file before stopping
|
||||
# No partial files — the mover completes what it's working on
|
||||
# 5. Wait 5 seconds — verify if stopped
|
||||
# 6. If still running → SIGKILL (forced)
|
||||
# Warning: partial files possible — same as a hard crash
|
||||
# 7. Final verify — error if still running
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
mover_stop.sh # stop mover with configured timeout
|
||||
mover_stop.sh --dry-run # show what would happen
|
||||
mover_stop.sh --status # show mover state (running, PID, start time)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔁 server_reboot.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Graceful reboot with pre-flight warnings, wall message, unRAID notification, VM
|
||||
graceful shutdown, then Docker and services stop, sync, reboot.
|
||||
|
||||
> Full documentation in `README-Tools.md` — `server_reboot.sh` section. This is a
|
||||
> quick reference.
|
||||
|
||||
---
|
||||
|
||||
### ── Shutdown Sequence ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Pre-flight warnings (warn not block):
|
||||
# rsync running, mover running, active Emby sessions
|
||||
# Warnings show in summary — you chose to reboot, these are for context
|
||||
# 2. Wall message + unRAID notification — MY_ID included
|
||||
# 3. Wait REBOOT_SLEEP seconds (default 30)
|
||||
# 4. virsh shutdown each VM → wait REBOOT_VM_WAIT seconds for graceful exit
|
||||
# 5. Stop libvirt (VM Manager)
|
||||
# 6. Stop Docker service
|
||||
# 7. sync — flush filesystem buffers
|
||||
# 8. /sbin/reboot
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
server_reboot.sh # reboot with 30s warning
|
||||
server_reboot.sh --dry-run # full sequence walkthrough without rebooting
|
||||
server_reboot.sh --status # show running processes that would be affected
|
||||
server_reboot.sh --reason="maintenance" # include reason in wall + notification
|
||||
server_reboot.sh --log # verbose per-step output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🛑 user_scripts_stop.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops all running User Script processes. Identifies by `/tmp/user.scripts` path
|
||||
signature. Shows script names not PIDs. SIGTERM → verify → SIGKILL with self-exclusion.
|
||||
|
||||
> Full documentation in `README-Tools.md` — `user_scripts_stop.sh` section.
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all — SIGTERM → SIGKILL if needed
|
||||
user_scripts_stop.sh --dry-run # show which scripts would be stopped, by name
|
||||
user_scripts_stop.sh --status # show running scripts with PID and runtime
|
||||
user_scripts_stop.sh --log # verbose per-process output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ STARTUP SEQUENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# master.conf — ARRAY_START_SCRIPTS (order matters)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
ARRAY_START_SCRIPTS=(
|
||||
# ── One-shot — run and exit ───────────────────────────────────────────────
|
||||
"unRAID_Essentials/inotify_tuning.sh" # FIRST — raise limits before
|
||||
# containers inherit old values
|
||||
"unRAID_Essentials/docker_syslog_filter.sh" # SECOND — before containers
|
||||
# create veth interfaces
|
||||
"unRAID_Essentials/php_fpm_max_children.sh" # before WebGUI serves requests
|
||||
"Transcodes/ramdisk_setup.sh" # before Emby starts transcoding
|
||||
"Docker_Essentials/docker_network_connect.sh" # before watchdogs check states
|
||||
|
||||
# ── Continuous — run until array stops ───────────────────────────────────
|
||||
"unRAID_Essentials/system_watchdog.sh" # before docker_watchdog —
|
||||
# writes state file docker_watchdog reads
|
||||
"Docker_Essentials/docker_watchdog.sh" # before failover — containers
|
||||
# must be healthy for failover decisions
|
||||
"Failover/failover.sh" # last — needs everything stable
|
||||
)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# array_start.sh is the ONLY "At Startup of Array" entry in User Scripts.
|
||||
# It launches everything above in order.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FULL SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# At Startup of Array — via array_start.sh:
|
||||
# inotify_tuning.sh
|
||||
# docker_syslog_filter.sh
|
||||
# php_fpm_max_children.sh
|
||||
# (ramdisk_setup.sh — in Transcodes/)
|
||||
# system_watchdog.sh (continuous)
|
||||
|
||||
# Every 10 minutes:
|
||||
*/10 * * * * webgui_restart.sh # silent when healthy — escalates when not
|
||||
|
||||
# Weekly — via weekly_sync_maintenance.sh:
|
||||
# clear_logs.sh # Sunday 2:30am via WEEKLY_MAINTENANCE_SCRIPTS
|
||||
|
||||
# Manual:
|
||||
# mover_stop.sh — before array ops that need mover stopped
|
||||
# server_reboot.sh — planned maintenance reboots
|
||||
# user_scripts_stop.sh — emergency script stop or pre-reboot cleanup
|
||||
```
|
||||
Reference in New Issue
Block a user