Gmer4Lfe
95151c2278
Move all watchdog scripts to a dedicated Watchdogs/ folder: Docker_Essentials/docker_watchdog.sh → Watchdogs/ unRAID_Essentials/system_watchdog.sh → Watchdogs/ unRAID_Essentials/resource_watchdog.sh → Watchdogs/ Orchestrators/watchdog_orchestrator.sh → Watchdogs/ Tools/watchdog_skip_list_manager.sh → Watchdogs/ Rename host config files: master_host1.conf → host1.conf master_host2.conf → host2.conf Update all references across the ecosystem: master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/ load_config.sh: host*.conf glob + all comments git_pull_execute.sh: sparse checkout glob + all comments Partnership/ssh_setup.sh: HOST_CONF path construction user_script_plug-in.sh: all script paths + per-host conf path common.sh, README.md, README-User_Script_Plug-in.md: comment refs All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
785 lines
25 KiB
Markdown
785 lines
25 KiB
Markdown
# ━━━━━ UNRAID ESSENTIALS — Manual ━━━━━
|
||
|
||
Configuration reference, operational procedures, and troubleshooting for
|
||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||
adding or reordering scripts at array start.
|
||
|
||
---
|
||
|
||
## ━━━ CONTENTS ━━━
|
||
|
||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||
- [system_watchdog.sh](#system_watchdogsh)
|
||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||
- [webgui_restart.sh](#webgui_restartsh)
|
||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
|
||
- [clear_logs.sh](#clear_logssh)
|
||
- [mover_stop.sh](#mover_stopsh)
|
||
- [rsync_stop.sh](#rsync_stopsh)
|
||
- [user_scripts_stop.sh](#user_scripts_stopsh)
|
||
- [server_reboot.sh](#server_rebootsh)
|
||
- [Full Configuration Reference](#full-configuration-reference)
|
||
- [Troubleshooting](#troubleshooting)
|
||
|
||
---
|
||
|
||
## Output Tiers
|
||
|
||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||
lines only appear when `--log` is passed.
|
||
|
||
**Daemon scripts** (`system_watchdog.sh`, `resource_watchdog.sh`, `webgui_restart.sh`):
|
||
run on every cycle. Without `--log`, only state transitions, warnings, errors, and
|
||
the clean-cycle conclusion line are visible. Per-check detail suppressed.
|
||
|
||
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
|
||
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
|
||
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
|
||
final summary are visible. Per-item detail suppressed.
|
||
|
||
---
|
||
|
||
## ARRAY_START_SCRIPTS Order
|
||
|
||
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
|
||
> scripts.** Getting it wrong causes subtle failures that don't show up
|
||
> immediately.
|
||
|
||
```bash
|
||
# master.conf
|
||
ARRAY_START_SCRIPTS=(
|
||
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
|
||
# any container starts. Containers inherit
|
||
# inotify limits at launch, not dynamically.
|
||
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
|
||
# created. If a container starts first, its
|
||
# veth creation is already in syslog.
|
||
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
|
||
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
|
||
...
|
||
"system_watchdog.sh" # LAST or near-last — starts background loop
|
||
)
|
||
```
|
||
|
||
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
|
||
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
|
||
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
|
||
|
||
Why docker_syslog_filter SECOND: The filter must be in place before any container
|
||
starts creating veth interfaces. The first container start after array start
|
||
generates veth messages — these will appear in syslog if the filter isn't active.
|
||
|
||
---
|
||
|
||
## system_watchdog.sh
|
||
|
||
### Three-Tier Response
|
||
|
||
The watchdog categorizes every failure into one of three tiers:
|
||
|
||
**Tier 1 — CRITICAL (immediate reboot, no strikes)**
|
||
| Condition | Threshold | Why immediate |
|
||
|-----------|-----------|---------------|
|
||
| Docker daemon unresponsive | N/A | Nothing can be healed; every docker command hangs |
|
||
| rootfs usage | SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) | SSH stops; state files fail silently |
|
||
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
|
||
| File descriptor exhaustion | SYS_WATCHDOG_FD_CRITICAL_PCT (95%) | New connections silently failing |
|
||
| /boot read-only unexpectedly | write test fails | Config writes silently failing |
|
||
|
||
**Tier 2 — URGENT (bypass strikes with OOM confirmation)**
|
||
|
||
RAM below MEM_GB AND OOM kills this cycle >= SYS_WATCHDOG_OOM_LIMIT.
|
||
Both conditions required — RAM alone without OOM uses the standard strike system.
|
||
OOM confirms the system is dying faster than watchdogs can heal.
|
||
|
||
**Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)**
|
||
| Check | Threshold |
|
||
|-------|-----------|
|
||
| Free RAM | MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB |
|
||
| Load average | SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count |
|
||
| CPU temperature | SYS_WATCHDOG_CPU_TEMP |
|
||
| Zombie processes | SYS_WATCHDOG_ZOMBIES |
|
||
| /var/log usage | SYS_WATCHDOG_VAR_LOG_PCT |
|
||
| /tmp usage | SYS_WATCHDOG_TMP_PCT |
|
||
| Array disk errors | mdstat error delta > 0 |
|
||
| NIC state | interface operstate != "up" |
|
||
| Required containers | containers in SYS_WATCHDOG_REQUIRED_CONTAINERS |
|
||
|
||
### RAM Tiers
|
||
|
||
```
|
||
MEM_WARN_GB (10GB) → warn + notify, no action
|
||
MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery
|
||
MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM)
|
||
MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart
|
||
```
|
||
|
||
At MEM_SHUTDOWN_GB, all containers NOT listed in
|
||
`SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` are stopped. Excluded containers by default:
|
||
NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr. Adjust in
|
||
master.conf for your critical services.
|
||
|
||
### Abort Conditions
|
||
|
||
These conditions prevent a reboot — running them would cause data loss:
|
||
|
||
```bash
|
||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # ZFS pool degraded/faulted
|
||
SYS_WATCHDOG_ABORT_ON_PARITY=true # Parity check/rebuild running
|
||
SYS_WATCHDOG_ABORT_ON_MOVER=true # Mover running
|
||
```
|
||
|
||
CRITICAL tier bypasses all abort conditions — an imminent crash outweighs
|
||
data safety concerns.
|
||
|
||
### Strike System
|
||
|
||
The strike count tracks consecutive failures. A single recovery resets strikes
|
||
to 0. The reboot only fires after SYS_WATCHDOG_STRIKES consecutive failures on
|
||
the same check — transient spikes (a brief load burst, a momentary RAM dip) don't
|
||
trigger reboots.
|
||
|
||
### Reboot Rate Limit
|
||
|
||
```bash
|
||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # window in hours
|
||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots within the window
|
||
```
|
||
|
||
If the server has rebooted SYS_WATCHDOG_MAX_REBOOTS times within the window,
|
||
system_watchdog stops rebooting and notifies instead. This prevents a boot loop
|
||
where the watchdog reboots → something crashes again immediately → reboot again.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
system_watchdog.sh # start continuous monitoring loop
|
||
system_watchdog.sh --dry-run # run detection logic without rebooting
|
||
system_watchdog.sh --status # show thresholds, current state, strike counts
|
||
system_watchdog.sh --log # verbose per-cycle output
|
||
```
|
||
|
||
### Verify Running
|
||
|
||
```bash
|
||
pgrep -a -f system_watchdog.sh
|
||
# Expected: shows PID and path
|
||
```
|
||
|
||
---
|
||
|
||
## resource_watchdog.sh
|
||
|
||
### Pressure Levels
|
||
|
||
```bash
|
||
# master.conf
|
||
RW_RAM_SOFT_GB=20 # Level 1 trigger — throttle downloaders
|
||
RW_RAM_MEDIUM_GB=15 # Level 2 trigger — throttle + pause containers
|
||
RW_RAM_HARD_GB=10 # Level 3 trigger — stop containers
|
||
RW_RAM_RECOVER_GB=25 # recover to this before un-stopping at level 3
|
||
|
||
RW_LOAD_SOFT_MULTIPLIER=2.0 # load > 2× cpu count = level 1
|
||
RW_LOAD_MEDIUM_MULTIPLIER=3.0 # load > 3× cpu count = level 2
|
||
|
||
RW_RECOVER_CYCLES=3 # consecutive under-threshold runs before de-escalating
|
||
```
|
||
|
||
### Per-Host Container Lists
|
||
|
||
```bash
|
||
# host1.conf
|
||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake") # paused at medium pressure
|
||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory") # stopped at hard pressure
|
||
```
|
||
|
||
Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
|
||
pressure level. Default includes: Emby, NginxProxyManager, Authelia, Mariadb, Redis.
|
||
|
||
### docker_watchdog Coordination
|
||
|
||
At level 3, resource_watchdog writes `mem_shutdown_active=true` to `RW_STATE_FILE`.
|
||
docker_watchdog.sh reads this flag each cycle and skips all container restart logic
|
||
while it is set. Without this coordination, docker_watchdog would immediately
|
||
restart containers that resource_watchdog just stopped to free RAM.
|
||
|
||
The flag is cleared when level 3 pressure resolves and containers are restarted.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
resource_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||
resource_watchdog.sh --dry-run # show what would be throttled/paused/stopped
|
||
resource_watchdog.sh --status # current level, active actions, recovery cycle count
|
||
resource_watchdog.sh --log # verbose per-check output
|
||
```
|
||
|
||
---
|
||
|
||
## webgui_restart.sh
|
||
|
||
### Escalation Logic
|
||
|
||
```
|
||
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
|
||
|
||
Not responding:
|
||
1. /etc/rc.d/rc.nginx restart
|
||
wait WEBGUI_NGINX_WAIT (15s) → recheck
|
||
→ recovered: notify, exit 0
|
||
|
||
2. /etc/rc.d/rc.php-fpm restart
|
||
wait WEBGUI_PHP_WAIT (10s) → recheck
|
||
→ recovered: notify, exit 0
|
||
|
||
3. /usr/local/sbin/emhttp stop && start
|
||
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
|
||
→ recovered: notify, exit 0
|
||
|
||
All three failed → notify warning, exit 1
|
||
```
|
||
|
||
### Configuration
|
||
|
||
```bash
|
||
WEBGUI_URL="http://localhost" # URL to check
|
||
WEBGUI_TIMEOUT=5 # curl timeout in seconds
|
||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
|
||
```
|
||
|
||
### WebGUI Frozen — Manual Recovery
|
||
|
||
```bash
|
||
# Check which services are running:
|
||
webgui_restart.sh --status
|
||
|
||
# Try manual restart sequence (same as the script):
|
||
/etc/rc.d/rc.nginx restart
|
||
# wait 15s, then:
|
||
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
|
||
|
||
# If nginx didn't fix it, php-fpm:
|
||
/etc/rc.d/rc.php-fpm restart
|
||
|
||
# If still down, emhttp:
|
||
/usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
|
||
|
||
# If all three failed:
|
||
server_reboot.sh --status # check for active sessions first
|
||
```
|
||
|
||
---
|
||
|
||
## inotify_tuning.sh
|
||
|
||
### What It Sets
|
||
|
||
```bash
|
||
INOTIFY_MAX_INSTANCES=1024 # max inotify fd objects per user (default: 128)
|
||
INOTIFY_MAX_WATCHES=1048576 # max watches shared across all users (default: 8192)
|
||
INOTIFY_MAX_QUEUED_EVENTS=32768 # max buffered events (default: 16384)
|
||
```
|
||
|
||
Verify current values:
|
||
|
||
```bash
|
||
inotify_tuning.sh --status
|
||
# Shows current vs target for each limit, active instance count, top consumers
|
||
```
|
||
|
||
### If Code-Server Shows "Unable to Watch for File Changes"
|
||
|
||
```bash
|
||
# 1. Verify limits are set:
|
||
sysctl fs.inotify.max_user_watches
|
||
# Expected: 1048576
|
||
|
||
# 2. Check total usage across all containers:
|
||
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l
|
||
|
||
# 3. If limits are set but Code-Server still shows the error:
|
||
docker restart Code-Server
|
||
# Running containers inherit limits at launch. Restart picks up the new values.
|
||
|
||
# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
|
||
inotify_tuning.sh --log
|
||
```
|
||
|
||
---
|
||
|
||
## php_fpm_max_children.sh
|
||
|
||
### What It Sets
|
||
|
||
```bash
|
||
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
|
||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||
```
|
||
|
||
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
|
||
The default of 4–8 saturates immediately under load on a busy server.
|
||
|
||
### Verify
|
||
|
||
```bash
|
||
php_fpm_max_children.sh --status
|
||
# Shows current value vs target, PHP-FPM worker count
|
||
|
||
# Manual verify:
|
||
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
|
||
# Expected: pm.max_children = 250
|
||
```
|
||
|
||
### If WebGUI Is Slow Despite the Setting
|
||
|
||
```bash
|
||
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
|
||
# Look at the webgui_restart.sh escalation — step 2 (php-fpm restart) is specifically
|
||
# for worker exhaustion. If webgui_restart.sh is regularly hitting step 2, the
|
||
# pm.max_children value may still be too low, or there's a PHP worker leak.
|
||
|
||
# Check running worker count:
|
||
pgrep -fc php-fpm
|
||
# Compare to pm.max_children — if equal, workers are saturated
|
||
|
||
# Increase if needed:
|
||
# master.conf: PHP_MAX_CHILDREN=350
|
||
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
|
||
```
|
||
|
||
---
|
||
|
||
## docker_syslog_filter.sh
|
||
|
||
### What It Creates
|
||
|
||
```
|
||
/etc/rsyslog.d/ignore-docker-veth.conf:
|
||
if ($msg contains "veth" or $msg contains "docker0") then {
|
||
stop
|
||
}
|
||
```
|
||
|
||
This drops any syslog message containing "veth" or "docker0" before it reaches
|
||
any output target, including the log file. The drop rule is applied at rsyslog
|
||
level — not at the log viewer level.
|
||
|
||
### Verify
|
||
|
||
```bash
|
||
docker_syslog_filter.sh --status
|
||
# Shows filter file content and rsyslog process state
|
||
|
||
# Manual verify:
|
||
cat /etc/rsyslog.d/ignore-docker-veth.conf
|
||
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"
|
||
|
||
# Test the filter is active (should produce no syslog output):
|
||
logger "test veth message"
|
||
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
|
||
```
|
||
|
||
### If Syslog Still Has Veth Noise
|
||
|
||
```bash
|
||
# 1. Verify filter file exists with correct content:
|
||
docker_syslog_filter.sh --status
|
||
|
||
# 2. If content differs — re-apply:
|
||
docker_syslog_filter.sh --log
|
||
|
||
# 3. Verify rsyslog is using the conf.d directory:
|
||
grep -r "IncludeConfig" /etc/rsyslog.conf
|
||
# Expected: IncludeConfig /etc/rsyslog.d/*.conf (or similar)
|
||
```
|
||
|
||
---
|
||
|
||
## clear_logs.sh
|
||
|
||
### Thresholds
|
||
|
||
```bash
|
||
LOG_MIN_SIZE_MB=10 # skip system log if under this — keep recent history
|
||
LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this
|
||
|
||
LOG_FILES=(
|
||
"/var/log/syslog"
|
||
"/var/log/messages"
|
||
"/var/log/dmesg"
|
||
)
|
||
```
|
||
|
||
### Why Truncation Not Deletion
|
||
|
||
unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
|
||
descriptor open and valid while emptying content — syslogd continues writing to
|
||
the same fd without interruption. Deleting the file would orphan the file
|
||
descriptor and syslog would stop writing until restarted.
|
||
|
||
### Identifying Large Docker Logs
|
||
|
||
```bash
|
||
clear_logs.sh --status
|
||
# Shows top 10 Docker logs by size, current size vs threshold
|
||
|
||
# Find the biggest log manually:
|
||
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
|
||
```
|
||
|
||
---
|
||
|
||
## mover_stop.sh
|
||
|
||
### Stop Sequence
|
||
|
||
```
|
||
1. Check if mover is running (pgrep "emhttp.*Mover") → exit cleanly if not
|
||
2. Wall message to all logged-in terminal users
|
||
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
|
||
4. SIGTERM — mover finishes its current file operation, then stops
|
||
5. Wait 5 seconds → verify stopped
|
||
6. SIGKILL if still running — forced stop, partial files possible
|
||
7. Final verify — error if still running after SIGKILL
|
||
```
|
||
|
||
SIGTERM first because the mover can finish the file it is currently moving,
|
||
leaving no partial copies split across cache and array. SIGKILL is a last resort.
|
||
|
||
### Configuration
|
||
|
||
```bash
|
||
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
|
||
```
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
mover_stop.sh # check and stop if running
|
||
mover_stop.sh --status # show current mover state and PID
|
||
mover_stop.sh --dry-run # show what would happen without stopping
|
||
```
|
||
|
||
---
|
||
|
||
## rsync_stop.sh
|
||
|
||
### Auto-Detection Logic
|
||
|
||
rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
|
||
is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.
|
||
|
||
**Default behavior (orchestrator detected):**
|
||
Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
|
||
next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.
|
||
|
||
**Default behavior (no orchestrator):**
|
||
Kills rsync directly (standalone rsync.sh run).
|
||
|
||
**--full-stop:**
|
||
Kills the orchestrator first, then kills rsync. Nothing continues after this.
|
||
Use when everything needs to stop immediately.
|
||
|
||
### Container Recovery
|
||
|
||
After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
|
||
for any that were stopped by the interrupted rsync session and restarts them.
|
||
Remote containers are left for docker_watchdog.sh to recover.
|
||
|
||
Skip container recovery with `--rsync-only` — used when called by other scripts
|
||
that handle recovery themselves.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
rsync_stop.sh # smart stop (auto-detect orchestrator)
|
||
rsync_stop.sh --full-stop # kill orchestrator + rsync
|
||
rsync_stop.sh --rsync-only # kill rsync, skip container recovery
|
||
rsync_stop.sh --status # show local and remote rsync state
|
||
rsync_stop.sh --dry-run # preview without changes
|
||
rsync_stop.sh --full-stop --dry-run # preview full stop
|
||
```
|
||
|
||
---
|
||
|
||
## user_scripts_stop.sh
|
||
|
||
### Process Identification
|
||
|
||
Scans `/proc/*/cmdline` for any process whose command line contains
|
||
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
|
||
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
|
||
what the script is named or how it was launched.
|
||
|
||
Script names are extracted from the path for display: you see which scripts are
|
||
being stopped, not just PIDs.
|
||
|
||
### Self-Exclusion
|
||
|
||
If this script is run via the User Scripts plugin, it would find its own PID in
|
||
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
|
||
from the kill list.
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
user_scripts_stop.sh # stop all User Script processes
|
||
user_scripts_stop.sh --status # show running scripts with names and elapsed time
|
||
user_scripts_stop.sh --dry-run # show what would be stopped
|
||
```
|
||
|
||
---
|
||
|
||
## server_reboot.sh
|
||
|
||
### Full Shutdown Sequence
|
||
|
||
```
|
||
1. Pre-flight checks (warn only — do not block):
|
||
- rsync running? → warn, suggest rsync_stop.sh first
|
||
- mover running? → warn, suggest mover_stop.sh first
|
||
- Emby sessions? → warn (active streams will be interrupted)
|
||
|
||
2. Wall message to all logged-in terminal users
|
||
|
||
3. unRAID dashboard notification
|
||
|
||
4. Wait REBOOT_SLEEP seconds (default: 30)
|
||
|
||
5. Graceful VM shutdown:
|
||
virsh shutdown <each VM> (ACPI signal — clean shutdown)
|
||
Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond
|
||
|
||
6. /etc/rc.d/rc.libvirt stop (VM Manager)
|
||
|
||
7. /etc/rc.d/rc.docker stop (all containers stop)
|
||
|
||
8. sync (flush filesystem buffers to disk)
|
||
|
||
9. /sbin/reboot
|
||
```
|
||
|
||
### Configuration
|
||
|
||
```bash
|
||
REBOOT_SLEEP=30 # seconds between warning and shutdown sequence
|
||
REBOOT_VM_WAIT=30 # seconds to wait for VMs to shut down gracefully
|
||
```
|
||
|
||
### Recommended Pre-Reboot Sequence
|
||
|
||
For a clean reboot when services are active:
|
||
|
||
```bash
|
||
rsync_stop.sh # stop any active rsync (smart mode)
|
||
mover_stop.sh # stop mover gracefully
|
||
server_reboot.sh --status # check what's still running
|
||
server_reboot.sh --reason="planned maintenance"
|
||
```
|
||
|
||
### Usage
|
||
|
||
```bash
|
||
server_reboot.sh # reboot with 30s warning
|
||
server_reboot.sh --dry-run # walk through without rebooting
|
||
server_reboot.sh --status # show what would be affected
|
||
server_reboot.sh --reason="disk work" # include reason in notification
|
||
```
|
||
|
||
---
|
||
|
||
## Full Configuration Reference
|
||
|
||
```bash
|
||
# master.conf
|
||
|
||
# ── System Watchdog ────────────────────────────────────────────────────────────
|
||
SYS_WATCHDOG_INTERVAL=300 # seconds between check cycles
|
||
SYS_WATCHDOG_STRIKES=3 # consecutive failures before reboot
|
||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2 # rate limit window
|
||
SYS_WATCHDOG_MAX_REBOOTS=3 # max reboots in window
|
||
SYS_WATCHDOG_OOM_LIMIT=3 # OOM kills/cycle for URGENT bypass
|
||
|
||
MEM_WARN_GB=10 # warn + notify
|
||
MEM_SHUTDOWN_GB=6 # stop non-essential containers
|
||
MEM_GB=4 # strike → reboot
|
||
MEM_RECOVER_GB=30 # recovery threshold
|
||
|
||
SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99 # Tier 1 trigger
|
||
SYS_WATCHDOG_FD_CRITICAL_PCT=95 # Tier 1 trigger
|
||
SYS_WATCHDOG_LOAD_MULTIPLIER=4 # Tier 3 — ×cpu_count
|
||
SYS_WATCHDOG_CPU_TEMP=85 # Tier 3 — Celsius
|
||
SYS_WATCHDOG_ZOMBIES=20 # Tier 3 — process count
|
||
SYS_WATCHDOG_VAR_LOG_PCT=80 # Tier 3 — percent full
|
||
SYS_WATCHDOG_TMP_PCT=85 # Tier 3 — percent full
|
||
|
||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
|
||
SYS_WATCHDOG_ABORT_ON_PARITY=true
|
||
SYS_WATCHDOG_ABORT_ON_MOVER=true
|
||
|
||
SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
|
||
"NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
|
||
)
|
||
SYS_WATCHDOG_REQUIRED_CONTAINERS=() # containers that must be running
|
||
|
||
# State files:
|
||
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
|
||
SYS_WATCHDOG_REBOOT_LOG="/tmp/sys_watchdog_reboots.db"
|
||
SYS_WATCHDOG_FAILED_FILE="/tmp/sys_watchdog_failed.db"
|
||
SYS_WATCHDOG_OOM_FILE="/tmp/sys_watchdog_oom.db"
|
||
|
||
# ── Resource Watchdog ──────────────────────────────────────────────────────────
|
||
RW_ENABLED=true
|
||
RW_STATE_FILE="/tmp/resource_watchdog_state.db"
|
||
|
||
RW_RAM_SOFT_GB=20
|
||
RW_RAM_MEDIUM_GB=15
|
||
RW_RAM_HARD_GB=10
|
||
RW_RAM_RECOVER_GB=25
|
||
|
||
RW_LOAD_SOFT_MULTIPLIER=2.0
|
||
RW_LOAD_MEDIUM_MULTIPLIER=3.0
|
||
RW_RECOVER_CYCLES=3
|
||
|
||
RW_SABNZBD_ENABLED=true
|
||
RW_SABNZBD_SPEED_SOFT="50M"
|
||
RW_SABNZBD_SPEED_MEDIUM="10M"
|
||
RW_QBIT_ENABLED=true
|
||
RW_QBIT_DL_SOFT=51200 # KB/s
|
||
RW_QBIT_DL_MEDIUM=10240
|
||
|
||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")
|
||
|
||
# ── Per-Host (host*.conf) ───────────────────────────────────────────────
|
||
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
|
||
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
|
||
HOST1_SABNZBD_URL="http://localhost:8080"
|
||
HOST1_SABNZBD_API_KEY="your-api-key"
|
||
HOST1_QBIT_URL="http://localhost:8090"
|
||
HOST1_QBIT_USERNAME="admin"
|
||
HOST1_QBIT_PASSWORD="your-password"
|
||
|
||
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
|
||
WEBGUI_URL="http://localhost"
|
||
WEBGUI_TIMEOUT=5
|
||
WEBGUI_NGINX_WAIT=15
|
||
WEBGUI_PHP_WAIT=10
|
||
WEBGUI_EMHTTP_WAIT=30
|
||
|
||
# ── inotify Tuning ─────────────────────────────────────────────────────────────
|
||
INOTIFY_MAX_INSTANCES=1024
|
||
INOTIFY_MAX_WATCHES=1048576
|
||
INOTIFY_MAX_QUEUED_EVENTS=32768
|
||
|
||
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
|
||
PHP_MAX_CHILDREN=250
|
||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||
|
||
# ── Syslog Filter ──────────────────────────────────────────────────────────────
|
||
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
||
|
||
# ── Log Cleaner ────────────────────────────────────────────────────────────────
|
||
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
|
||
LOG_MIN_SIZE_MB=10
|
||
LOG_DOCKER_MAX_MB=100
|
||
|
||
# ── Mover Stop ─────────────────────────────────────────────────────────────────
|
||
MOVER_STOP_TIMEOUT=30
|
||
|
||
# ── Server Reboot ──────────────────────────────────────────────────────────────
|
||
REBOOT_SLEEP=30
|
||
REBOOT_VM_WAIT=30
|
||
```
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### system_watchdog.sh Not Starting
|
||
|
||
```bash
|
||
# Check if already running (acquire_lock prevents second instance):
|
||
pgrep -a -f system_watchdog.sh
|
||
|
||
# Check state file permissions:
|
||
ls -la /tmp/sys_watchdog_*.db
|
||
|
||
# Run with --log to see startup:
|
||
system_watchdog.sh --log
|
||
```
|
||
|
||
### system_watchdog Rebooted Unexpectedly
|
||
|
||
```bash
|
||
# Check reboot log:
|
||
cat /tmp/sys_watchdog_reboots.db
|
||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||
|
||
# Check what condition triggered it:
|
||
# Look in /var/log/syslog for "system_watchdog" near the reboot time
|
||
grep "system_watchdog" /var/log/syslog | tail -20
|
||
```
|
||
|
||
### resource_watchdog Paused Containers It Shouldn't Have
|
||
|
||
```bash
|
||
# Check current state:
|
||
resource_watchdog.sh --status
|
||
|
||
# Add the container to RW_CRITICAL_CONTAINERS in master.conf:
|
||
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")
|
||
|
||
# Un-pause manually if needed:
|
||
docker unpause MyContainer
|
||
```
|
||
|
||
### rsync_stop Killed the Wrong Thing
|
||
|
||
If --full-stop killed an orchestrator you didn't intend to kill:
|
||
|
||
```bash
|
||
# Next time use default mode (no --full-stop) to kill only rsync subprocess.
|
||
# To verify what would be killed before running:
|
||
rsync_stop.sh --status # shows running rsync and detected orchestrators
|
||
rsync_stop.sh --dry-run # shows smart mode decision
|
||
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
|
||
```
|
||
|
||
### WebGUI Recovery After All Three Steps Failed
|
||
|
||
```bash
|
||
# Check if processes are running:
|
||
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
|
||
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
|
||
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
|
||
|
||
# Check recent nginx errors:
|
||
cat /var/log/nginx/error.log | tail -20
|
||
|
||
# Check emhttp log:
|
||
tail -20 /var/log/syslog | grep emhttp
|
||
|
||
# Last resort — reboot:
|
||
server_reboot.sh --reason="WebGUI unrecoverable"
|
||
```
|
||
|
||
### PHP-FPM Config Not Found After unRAID Update
|
||
|
||
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
|
||
errors with "config file not found":
|
||
|
||
```bash
|
||
# Find the new config path:
|
||
find /etc -name "www.conf" 2>/dev/null
|
||
|
||
# Update PHP_CONF in master.conf:
|
||
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
|
||
|
||
# Verify with:
|
||
php_fpm_max_children.sh --status
|
||
```
|