12 KiB
unRAID Essentials
System-level scripts that act on the unRAID server itself — not containers, not media, not monitoring. These scripts keep the server healthy, respond to problems, and handle graceful shutdown and restart sequences.
Monitors/ — observes and reports
Docker_Essentials/ — acts on containers
unRAID_Essentials/ — acts on the server itself (this folder)
Most scripts in this folder are either scheduled at array start or run on a weekly maintenance schedule. A few are utilities called manually or by other scripts. All support --dry-run.
Scripts
system_watchdog.sh
The last line of defense. Reboots the system cleanly when it is about to become unstable. Works alongside docker_watchdog.sh — containers are the first line of healing, system reboot is the last resort.
# Scheduled as: */15 * * * * (every 15 minutes)
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh
Relationship with docker_watchdog.sh:
docker_watchdog.sh — container level, tries to self-heal first
memory limits, CPU, HTTP checks, restarts
system_watchdog.sh — system level, acts when healing has failed
reboots when the server itself is unstable
What it checks (all individually toggleable in Master.conf):
| Check | Threshold | Why It Matters |
|---|---|---|
| rootfs usage | 95% | Fills rapidly when array is down — crash imminent |
| /var/log usage | 95% | Log spam filling rootfs — indicates something broken |
| Free RAM | 4GB | Critically low RAM means OOM kills or swap imminent |
| ZFS ARC pinned | 98% | ARC not releasing after reclaim — memory stuck |
| CPU temperature | 95°C | Sustained tjmax causes throttling or kernel panic |
| Load average | cores × 3 | Sustained high load — something stuck or runaway |
| Zombie processes | 50 | Large zombie count — serious process management failure |
| Docker daemon | responsive | Unresponsive daemon means containers cannot be managed |
| Required containers | running | Stopped containers the watchdog couldn't recover |
Strike system:
Checks use a strike system — a single spike doesn't trigger a reboot. The threshold must be hit on consecutive cycles. SYS_WATCHDOG_STRIKE_LIMIT=2 means two consecutive 15-minute cycles above the threshold before acting. Single spikes are ignored.
Abort conditions:
Some conditions prevent a reboot even if thresholds are hit:
| Condition | Default | Reason |
|---|---|---|
| ZFS pool unhealthy | abort | Rebooting with bad pool risks data loss |
| Parity running | reboot anyway | Aborting parity beats crashing mid-check |
| Mover running | reboot anyway | Aborting move beats crashing mid-move |
Set true to abort reboot if condition is active. Set false to reboot regardless. Philosophy: a graceful reboot before a crash is always better than a hard crash mid-operation.
Reboot loop protection:
Tracks reboot timestamps in a persistent log on /boot/ — survives reboots. If the server reboots SYS_WATCHDOG_REBOOT_LIMIT times within SYS_WATCHDOG_REBOOT_WINDOW_HRS hours it shuts down instead. A reboot loop means something is fundamentally broken that rebooting is not fixing — shutting down prevents hardware damage and gives you time to investigate.
State files:
/tmp/system_watchdog_state.db — strike counts (resets on reboot)
/boot/config/system_watchdog_failed.db — container skip list (persistent)
/boot/config/system_watchdog_reboots.db — reboot timestamps (persistent)
webgui_restart.sh
Monitors the unRAID WebGUI and restarts it automatically if unresponsive.
# Scheduled as: */10 * * * * (every 10 minutes)
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh
Escalation path:
1. curl WebGUI → unresponsive
2. Restart nginx → wait WEBGUI_NGINX_WAIT seconds → recheck
3. Still unresponsive → restart emhttp → wait WEBGUI_EMHTTP_WAIT seconds → recheck
4. Still unresponsive → notify warning — manual intervention needed
Why emhttp is more disruptive:
nginx is the web server layer — restarting it is fast and clean. emhttp is the core unRAID management daemon — it manages the array, Docker, VMs, and everything else. Restarting it takes longer but recovers cleanly. The escalation path tries the less disruptive option first.
A notification is sent on any restart — nginx or emhttp — so you know what happened and when.
Configuration:
WEBGUI_URL="http://localhost" # adjust if running non-standard port
WEBGUI_TIMEOUT=5 # seconds before curl gives up
WEBGUI_NGINX_WAIT=15 # wait after nginx restart before recheck
WEBGUI_EMHTTP_WAIT=30 # wait after emhttp restart — takes longer
docker_syslog_filter.sh
Suppresses noisy Docker network interface messages from the unRAID syslog.
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh
The problem it solves:
Every time Docker starts a container it creates virtual network interfaces (veth devices). Every time a container stops, they're removed. Each creation and removal generates syslog entries. On a server with 50+ containers starting at array start this creates hundreds of lines of noise that buries real log messages.
The filter creates an rsyslog configuration file that suppresses these specific messages and restarts rsyslog to apply it. The filter file is recreated on every array start — rsyslog configuration doesn't survive reboots on unRAID.
Configuration:
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
php_fpm_max_children.sh
Sets the PHP-FPM pm.max_children value to allow more concurrent requests to the unRAID WebGUI.
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/php_fpm_max_children.sh
Why this is needed:
unRAID's WebGUI is served via PHP-FPM. The default pm.max_children value is conservative. On a server with many users, plugins, or automated tools hitting the API simultaneously, the default value can cause requests to queue or time out. Increasing it allows more concurrent PHP processes.
The setting does not survive reboots — PHP-FPM configuration is reset on each boot. Running this at array start ensures it's always applied.
Configuration:
PHP_CONF="/etc/php-fpm.d/www.conf"
PHP_MAX_CHILDREN=250 # set based on available RAM
# each PHP worker uses ~30-50MB
# 250 workers × 40MB = ~10GB worst case
Set PHP_MAX_CHILDREN based on your available RAM. Higher values allow more concurrency but use more memory if all workers are active simultaneously.
clear_logs.sh
Clears unRAID system log files and Docker container logs.
# Scheduled as: 0 5 * * 0 (Sunday 5am weekly)
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh
Why weekly log clearing:
unRAID system logs live in RAM (/var/log/) — they don't persist across reboots. However on a stable server that runs for weeks without rebooting, these logs grow continuously. Docker container logs can grow particularly large if a container is verbose. Both can fill rootfs if left unchecked.
Weekly clearing keeps rootfs clean without being too aggressive. If you need to investigate a problem the week's logs are still available.
Configuration:
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
# Docker container logs are cleared automatically — no configuration needed
mover_stop.sh
Safely stops the unRAID mover with a warning delay before terminating.
# Run manually when needed — not scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/mover_stop.sh
When you need this:
The mover moves files from the cache pool to the array. If you need to stop it mid-run — before an array stop, before maintenance, or because it's been running too long — killing it directly can leave files in an inconsistent state. This script warns first and gives the mover time to finish its current file operation cleanly.
Configuration:
MOVER_STOP_TIMEOUT=300 # seconds to wait before sending SIGTERM
# gives mover time to finish current file
rsync_stop.sh
Stops all running rsync processes on both local and remote servers.
# Run manually when needed — not scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/rsync_stop.sh
When you need this:
An rsync job may need to be stopped — before an array stop, because it's consuming too much bandwidth, or because it started at the wrong time. Simply killing rsync can leave containers stopped on the remote server (since rsync.sh stops containers before syncing and restarts them after).
This script:
- Kills rsync processes locally
- Kills rsync processes on the remote via SSH
- Checks all configured profile containers locally
- Restarts any containers that were left stopped by the interrupted rsync
The remote is left in whatever container state it was in — the remote server's watchdog handles recovery on its own.
server_reboot.sh
Gracefully reboots the unRAID server with a warning delay.
# Run manually when needed — not typically scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/server_reboot.sh
Sequence:
1. Broadcast warning to all logged-in users
2. Wait REBOOT_SLEEP seconds
3. Stop Docker gracefully
4. Stop VM Manager gracefully
5. Issue reboot
The warning gives users time to finish what they are doing — saving files in VS Code Server, finishing a download, wrapping up a session. The Docker and VM stop ensures containers and VMs shut down cleanly rather than being hard-killed by the reboot.
Configuration:
REBOOT_SLEEP=300 # seconds of warning before reboot (default: 5 minutes)
Note: system_watchdog.sh calls this script automatically when thresholds are exceeded. You can also call it manually for planned maintenance.
user_script_stop.sh
Stops all running User Scripts plugin jobs.
# Run manually when needed — not typically scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/user_script_stop.sh
When you need this:
Before stopping the array, before a reboot, or when a script has hung and needs to be cleared. The User Scripts plugin runs scripts in /tmp/user.scripts/ — this script identifies all running processes with that path signature and terminates them cleanly.
Useful before server_reboot.sh to ensure no scripts are mid-execution when the reboot happens.
Startup Sequence
The recommended array start sequence for scripts in this folder:
# At Startup of Array — in this order
ramdisk_setup.sh # Transcodes/ — creates ramdisk before anything uses it
docker_syslog_filter.sh # suppress veth noise before containers start
php_fpm_max_children.sh # WebGUI performance before anyone accesses it
docker_network_connect.sh # Docker_Essentials/ — connect containers to networks
These run once at array start. The order matters — ramdisk before containers, filter before logs fill with noise, PHP config before WebGUI requests arrive.
Scheduled Maintenance Summary
# At Startup of Array
docker_syslog_filter.sh
php_fpm_max_children.sh
# Every 10 minutes
*/10 * * * * webgui_restart.sh
# Every 15 minutes
*/15 * * * * system_watchdog.sh
# Weekly — Sunday morning
0 5 * * 0 clear_logs.sh
--dry-run Support
Every script in this folder supports --dry-run. Always test before scheduling:
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh --dry-run
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --dry-run
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/server_reboot.sh --dry-run
server_reboot.sh --dry-run is particularly useful — it walks through the entire shutdown sequence, shows what would be stopped, and exits without rebooting.