# unRAID Essentials

System-level scripts that act on the unRAID server itself — not containers, not media, not monitoring. These scripts keep the server healthy, respond to problems, and handle graceful shutdown and restart sequences.

```
Monitors/            — observes and reports
Docker_Essentials/   — acts on containers
unRAID_Essentials/   — acts on the server itself (this folder)
```

Most scripts in this folder are either scheduled at array start or run on a weekly maintenance schedule. A few are utilities called manually or by other scripts. All support `--dry-run`.

---

## Scripts

### `system_watchdog.sh`

**The last line of defense.** Reboots the system cleanly when it is about to become unstable. Works alongside `docker_watchdog.sh` — containers are the first line of healing, system reboot is the last resort.

```bash
# Scheduled as: */15 * * * *  (every 15 minutes)
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh
```

**Relationship with docker_watchdog.sh:**
```
docker_watchdog.sh   — container level, tries to self-heal first
                       memory limits, CPU, HTTP checks, restarts
                       
system_watchdog.sh   — system level, acts when healing has failed
                       reboots when the server itself is unstable
```

**What it checks** (all individually toggleable in `Master.conf`):

| Check | Threshold | Why It Matters |
|-------|-----------|----------------|
| rootfs usage | 95% | Fills rapidly when array is down — crash imminent |
| /var/log usage | 95% | Log spam filling rootfs — indicates something broken |
| Free RAM | 4GB | Critically low RAM means OOM kills or swap imminent |
| ZFS ARC pinned | 98% | ARC not releasing after reclaim — memory stuck |
| CPU temperature | 95°C | Sustained tjmax causes throttling or kernel panic |
| Load average | cores × 3 | Sustained high load — something stuck or runaway |
| Zombie processes | 50 | Large zombie count — serious process management failure |
| Docker daemon | responsive | Unresponsive daemon means containers cannot be managed |
| Required containers | running | Stopped containers the watchdog couldn't recover |

**Strike system:**

Checks use a strike system — a single spike doesn't trigger a reboot. The threshold must be hit on consecutive cycles. `SYS_WATCHDOG_STRIKE_LIMIT=2` means two consecutive 15-minute cycles above the threshold before acting. Single spikes are ignored.

**Abort conditions:**

Some conditions prevent a reboot even if thresholds are hit:

| Condition | Default | Reason |
|-----------|---------|--------|
| ZFS pool unhealthy | abort | Rebooting with bad pool risks data loss |
| Parity running | reboot anyway | Aborting parity beats crashing mid-check |
| Mover running | reboot anyway | Aborting move beats crashing mid-move |

Set `true` to abort reboot if condition is active. Set `false` to reboot regardless. Philosophy: a graceful reboot before a crash is always better than a hard crash mid-operation.

**Reboot loop protection:**

Tracks reboot timestamps in a persistent log on `/boot/` — survives reboots. If the server reboots `SYS_WATCHDOG_REBOOT_LIMIT` times within `SYS_WATCHDOG_REBOOT_WINDOW_HRS` hours it shuts down instead. A reboot loop means something is fundamentally broken that rebooting is not fixing — shutting down prevents hardware damage and gives you time to investigate.

**State files:**
```
/tmp/system_watchdog_state.db        — strike counts (resets on reboot)
/boot/config/system_watchdog_failed.db — container skip list (persistent)
/boot/config/system_watchdog_reboots.db — reboot timestamps (persistent)
```

---

### `webgui_restart.sh`

Monitors the unRAID WebGUI and restarts it automatically if unresponsive.

```bash
# Scheduled as: */10 * * * *  (every 10 minutes)
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh
```

**Escalation path:**
```
1. curl WebGUI → unresponsive
2. Restart nginx → wait WEBGUI_NGINX_WAIT seconds → recheck
3. Still unresponsive → restart emhttp → wait WEBGUI_EMHTTP_WAIT seconds → recheck
4. Still unresponsive → notify warning — manual intervention needed
```

**Why emhttp is more disruptive:**

`nginx` is the web server layer — restarting it is fast and clean. `emhttp` is the core unRAID management daemon — it manages the array, Docker, VMs, and everything else. Restarting it takes longer but recovers cleanly. The escalation path tries the less disruptive option first.

A notification is sent on any restart — nginx or emhttp — so you know what happened and when.

**Configuration:**
```bash
WEBGUI_URL="http://localhost"   # adjust if running non-standard port
WEBGUI_TIMEOUT=5                # seconds before curl gives up
WEBGUI_NGINX_WAIT=15            # wait after nginx restart before recheck
WEBGUI_EMHTTP_WAIT=30           # wait after emhttp restart — takes longer
```

---

### `docker_syslog_filter.sh`

Suppresses noisy Docker network interface messages from the unRAID syslog.

```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh
```

**The problem it solves:**

Every time Docker starts a container it creates virtual network interfaces (`veth` devices). Every time a container stops, they're removed. Each creation and removal generates syslog entries. On a server with 50+ containers starting at array start this creates hundreds of lines of noise that buries real log messages.

The filter creates an rsyslog configuration file that suppresses these specific messages and restarts rsyslog to apply it. The filter file is recreated on every array start — rsyslog configuration doesn't survive reboots on unRAID.

**Configuration:**
```bash
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
```

---

### `php_fpm_max_children.sh`

Sets the PHP-FPM `pm.max_children` value to allow more concurrent requests to the unRAID WebGUI.

```bash
# Scheduled as: At Startup of Array
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/php_fpm_max_children.sh
```

**Why this is needed:**

unRAID's WebGUI is served via PHP-FPM. The default `pm.max_children` value is conservative. On a server with many users, plugins, or automated tools hitting the API simultaneously, the default value can cause requests to queue or time out. Increasing it allows more concurrent PHP processes.

The setting does not survive reboots — PHP-FPM configuration is reset on each boot. Running this at array start ensures it's always applied.

**Configuration:**
```bash
PHP_CONF="/etc/php-fpm.d/www.conf"
PHP_MAX_CHILDREN=250    # set based on available RAM
                        # each PHP worker uses ~30-50MB
                        # 250 workers × 40MB = ~10GB worst case
```

Set `PHP_MAX_CHILDREN` based on your available RAM. Higher values allow more concurrency but use more memory if all workers are active simultaneously.

---

### `clear_logs.sh`

Clears unRAID system log files and Docker container logs.

```bash
# Scheduled as: 0 5 * * 0  (Sunday 5am weekly)
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh
```

**Why weekly log clearing:**

unRAID system logs live in RAM (`/var/log/`) — they don't persist across reboots. However on a stable server that runs for weeks without rebooting, these logs grow continuously. Docker container logs can grow particularly large if a container is verbose. Both can fill rootfs if left unchecked.

Weekly clearing keeps rootfs clean without being too aggressive. If you need to investigate a problem the week's logs are still available.

**Configuration:**
```bash
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
# Docker container logs are cleared automatically — no configuration needed
```

---

### `mover_stop.sh`

Safely stops the unRAID mover with a warning delay before terminating.

```bash
# Run manually when needed — not scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/mover_stop.sh
```

**When you need this:**

The mover moves files from the cache pool to the array. If you need to stop it mid-run — before an array stop, before maintenance, or because it's been running too long — killing it directly can leave files in an inconsistent state. This script warns first and gives the mover time to finish its current file operation cleanly.

**Configuration:**
```bash
MOVER_STOP_TIMEOUT=300  # seconds to wait before sending SIGTERM
                         # gives mover time to finish current file
```

---

### `rsync_stop.sh`

Stops all running rsync processes on both local and remote servers.

```bash
# Run manually when needed — not scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/rsync_stop.sh
```

**When you need this:**

An rsync job may need to be stopped — before an array stop, because it's consuming too much bandwidth, or because it started at the wrong time. Simply killing rsync can leave containers stopped on the remote server (since rsync.sh stops containers before syncing and restarts them after).

This script:
1. Kills rsync processes locally
2. Kills rsync processes on the remote via SSH
3. Checks all configured profile containers locally
4. Restarts any containers that were left stopped by the interrupted rsync

The remote is left in whatever container state it was in — the remote server's watchdog handles recovery on its own.

---

### `server_reboot.sh`

Gracefully reboots the unRAID server with a warning delay.

```bash
# Run manually when needed — not typically scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/server_reboot.sh
```

**Sequence:**
```
1. Broadcast warning to all logged-in users
2. Wait REBOOT_SLEEP seconds
3. Stop Docker gracefully
4. Stop VM Manager gracefully
5. Issue reboot
```

The warning gives users time to finish what they are doing — saving files in VS Code Server, finishing a download, wrapping up a session. The Docker and VM stop ensures containers and VMs shut down cleanly rather than being hard-killed by the reboot.

**Configuration:**
```bash
REBOOT_SLEEP=300    # seconds of warning before reboot (default: 5 minutes)
```

**Note:** `system_watchdog.sh` calls this script automatically when thresholds are exceeded. You can also call it manually for planned maintenance.

---

### `user_script_stop.sh`

Stops all running User Scripts plugin jobs.

```bash
# Run manually when needed — not typically scheduled
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/user_script_stop.sh
```

**When you need this:**

Before stopping the array, before a reboot, or when a script has hung and needs to be cleared. The User Scripts plugin runs scripts in `/tmp/user.scripts/` — this script identifies all running processes with that path signature and terminates them cleanly.

Useful before `server_reboot.sh` to ensure no scripts are mid-execution when the reboot happens.

---

## Startup Sequence

The recommended array start sequence for scripts in this folder:

```bash
# At Startup of Array — in this order
ramdisk_setup.sh            # Transcodes/ — creates ramdisk before anything uses it
docker_syslog_filter.sh     # suppress veth noise before containers start
php_fpm_max_children.sh     # WebGUI performance before anyone accesses it
docker_network_connect.sh   # Docker_Essentials/ — connect containers to networks
```

These run once at array start. The order matters — ramdisk before containers, filter before logs fill with noise, PHP config before WebGUI requests arrive.

---

## Scheduled Maintenance Summary

```bash
# At Startup of Array
docker_syslog_filter.sh
php_fpm_max_children.sh

# Every 10 minutes
*/10 * * * *    webgui_restart.sh

# Every 15 minutes
*/15 * * * *    system_watchdog.sh

# Weekly — Sunday morning
0 5 * * 0       clear_logs.sh
```

---

## --dry-run Support

Every script in this folder supports `--dry-run`. Always test before scheduling:

```bash
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh --dry-run
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --dry-run
/mnt/user/appdata/unraid_scripts/unRAID_Essentials/server_reboot.sh --dry-run
```

`server_reboot.sh --dry-run` is particularly useful — it walks through the entire shutdown sequence, shows what would be stopped, and exits without rebooting.