# ━━━━━ UNRAID ESSENTIALS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.

---

## ━━━ CONTENTS ━━━

- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [system_watchdog.sh](#system_watchdogsh)
- [resource_watchdog.sh](#resource_watchdogsh)
- [webgui_restart.sh](#webgui_restartsh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
- [clear_logs.sh](#clear_logssh)
- [mover_stop.sh](#mover_stopsh)
- [rsync_stop.sh](#rsync_stopsh)
- [user_scripts_stop.sh](#user_scripts_stopsh)
- [server_reboot.sh](#server_rebootsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)

---

## Output Tiers

All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.

**Daemon scripts** (`system_watchdog.sh`, `resource_watchdog.sh`, `webgui_restart.sh`):
run on every cycle. Without `--log`, only state transitions, warnings, errors, and
the clean-cycle conclusion line are visible. Per-check detail suppressed.

**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.

---

## ARRAY_START_SCRIPTS Order

> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
> scripts.** Getting it wrong causes subtle failures that don't show up
> immediately.

```bash
# master.conf
ARRAY_START_SCRIPTS=(
    "inotify_tuning.sh"           # 1 — FIRST: kernel limits must be set before
                                   #    any container starts. Containers inherit
                                   #    inotify limits at launch, not dynamically.
    "docker_syslog_filter.sh"     # 2 — SECOND: before any veth interfaces are
                                   #    created. If a container starts first, its
                                   #    veth creation is already in syslog.
    "php_fpm_max_children.sh"     # 3 — before WebGUI is under load
    "ramdisk_setup.sh"            # (from Transcodes/) before Emby starts
    ...
    "system_watchdog.sh"          # LAST or near-last — starts background loop
)
```

Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.

Why docker_syslog_filter SECOND: The filter must be in place before any container
starts creating veth interfaces. The first container start after array start
generates veth messages — these will appear in syslog if the filter isn't active.

---

## system_watchdog.sh

### Three-Tier Response

The watchdog categorizes every failure into one of three tiers:

**Tier 1 — CRITICAL (immediate reboot, no strikes)**
| Condition | Threshold | Why immediate |
|-----------|-----------|---------------|
| Docker daemon unresponsive | N/A | Nothing can be healed; every docker command hangs |
| rootfs usage | SYS_WATCHDOG_ROOTFS_CRITICAL_PCT (99%) | SSH stops; state files fail silently |
| Kernel oops/BUG in dmesg | delta > 0 | Kernel running with corrupted state |
| File descriptor exhaustion | SYS_WATCHDOG_FD_CRITICAL_PCT (95%) | New connections silently failing |
| /boot read-only unexpectedly | write test fails | Config writes silently failing |

**Tier 2 — URGENT (bypass strikes with OOM confirmation)**

RAM below MEM_GB AND OOM kills this cycle >= SYS_WATCHDOG_OOM_LIMIT.
Both conditions required — RAM alone without OOM uses the standard strike system.
OOM confirms the system is dying faster than watchdogs can heal.

**Tier 3 — STANDARD (SYS_WATCHDOG_STRIKES consecutive failures → reboot)**
| Check | Threshold |
|-------|-----------|
| Free RAM | MEM_WARN_GB → MEM_SHUTDOWN_GB → MEM_GB |
| Load average | SYS_WATCHDOG_LOAD_MULTIPLIER × cpu_count |
| CPU temperature | SYS_WATCHDOG_CPU_TEMP |
| Zombie processes | SYS_WATCHDOG_ZOMBIES |
| /var/log usage | SYS_WATCHDOG_VAR_LOG_PCT |
| /tmp usage | SYS_WATCHDOG_TMP_PCT |
| Array disk errors | mdstat error delta > 0 |
| NIC state | interface operstate != "up" |
| Required containers | containers in SYS_WATCHDOG_REQUIRED_CONTAINERS |

### RAM Tiers

```
MEM_WARN_GB (10GB)      → warn + notify, no action
MEM_SHUTDOWN_GB (6GB)   → stop non-essential containers, wait for recovery
MEM_GB (4GB)            → strike → reboot (URGENT bypass with OOM)
MEM_RECOVER_GB (30GB)   → RAM must reach this before stopped containers restart
```

At MEM_SHUTDOWN_GB, all containers NOT listed in
`SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` are stopped. Excluded containers by default:
NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr. Adjust in
master.conf for your critical services.

### Abort Conditions

These conditions prevent a reboot — running them would cause data loss:

```bash
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true   # ZFS pool degraded/faulted
SYS_WATCHDOG_ABORT_ON_PARITY=true          # Parity check/rebuild running
SYS_WATCHDOG_ABORT_ON_MOVER=true           # Mover running
```

CRITICAL tier bypasses all abort conditions — an imminent crash outweighs
data safety concerns.

### Strike System

The strike count tracks consecutive failures. A single recovery resets strikes
to 0. The reboot only fires after SYS_WATCHDOG_STRIKES consecutive failures on
the same check — transient spikes (a brief load burst, a momentary RAM dip) don't
trigger reboots.

### Reboot Rate Limit

```bash
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2   # window in hours
SYS_WATCHDOG_MAX_REBOOTS=3         # max reboots within the window
```

If the server has rebooted SYS_WATCHDOG_MAX_REBOOTS times within the window,
system_watchdog stops rebooting and notifies instead. This prevents a boot loop
where the watchdog reboots → something crashes again immediately → reboot again.

### Usage

```bash
system_watchdog.sh              # start continuous monitoring loop
system_watchdog.sh --dry-run    # run detection logic without rebooting
system_watchdog.sh --status     # show thresholds, current state, strike counts
system_watchdog.sh --log        # verbose per-cycle output
```

### Verify Running

```bash
pgrep -a -f system_watchdog.sh
# Expected: shows PID and path
```

---

## resource_watchdog.sh

### Pressure Levels

```bash
# master.conf
RW_RAM_SOFT_GB=20           # Level 1 trigger — throttle downloaders
RW_RAM_MEDIUM_GB=15         # Level 2 trigger — throttle + pause containers
RW_RAM_HARD_GB=10           # Level 3 trigger — stop containers
RW_RAM_RECOVER_GB=25        # recover to this before un-stopping at level 3

RW_LOAD_SOFT_MULTIPLIER=2.0    # load > 2× cpu count = level 1
RW_LOAD_MEDIUM_MULTIPLIER=3.0  # load > 3× cpu count = level 2

RW_RECOVER_CYCLES=3         # consecutive under-threshold runs before de-escalating
```

### Per-Host Container Lists

```bash
# host1.conf
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")        # paused at medium pressure
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")    # stopped at hard pressure
```

Containers in `RW_CRITICAL_CONTAINERS` are never paused or stopped regardless of
pressure level. Default includes: Emby, NginxProxyManager, Authelia, Mariadb, Redis.

### docker_watchdog Coordination

At level 3, resource_watchdog writes `mem_shutdown_active=true` to `RW_STATE_FILE`.
docker_watchdog.sh reads this flag each cycle and skips all container restart logic
while it is set. Without this coordination, docker_watchdog would immediately
restart containers that resource_watchdog just stopped to free RAM.

The flag is cleared when level 3 pressure resolves and containers are restarted.

### Usage

```bash
resource_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
resource_watchdog.sh --dry-run  # show what would be throttled/paused/stopped
resource_watchdog.sh --status   # current level, active actions, recovery cycle count
resource_watchdog.sh --log      # verbose per-check output
```

---

## webgui_restart.sh

### Escalation Logic

```
curl $WEBGUI_URL → 200 OK → exit 0 (silent)

Not responding:
  1. /etc/rc.d/rc.nginx restart
     wait WEBGUI_NGINX_WAIT (15s) → recheck
     → recovered: notify, exit 0

  2. /etc/rc.d/rc.php-fpm restart
     wait WEBGUI_PHP_WAIT (10s) → recheck
     → recovered: notify, exit 0

  3. /usr/local/sbin/emhttp stop && start
     wait WEBGUI_EMHTTP_WAIT (30s) → recheck
     → recovered: notify, exit 0

  All three failed → notify warning, exit 1
```

### Configuration

```bash
WEBGUI_URL="http://localhost"   # URL to check
WEBGUI_TIMEOUT=5                # curl timeout in seconds
WEBGUI_NGINX_WAIT=15            # seconds after nginx restart before recheck
WEBGUI_PHP_WAIT=10              # seconds after php-fpm restart before recheck
WEBGUI_EMHTTP_WAIT=30           # seconds after emhttp restart before recheck
```

### WebGUI Frozen — Manual Recovery

```bash
# Check which services are running:
webgui_restart.sh --status

# Try manual restart sequence (same as the script):
/etc/rc.d/rc.nginx restart
# wait 15s, then:
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"

# If nginx didn't fix it, php-fpm:
/etc/rc.d/rc.php-fpm restart

# If still down, emhttp:
/usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start

# If all three failed:
server_reboot.sh --status   # check for active sessions first
```

---

## inotify_tuning.sh

### What It Sets

```bash
INOTIFY_MAX_INSTANCES=1024       # max inotify fd objects per user (default: 128)
INOTIFY_MAX_WATCHES=1048576      # max watches shared across all users (default: 8192)
INOTIFY_MAX_QUEUED_EVENTS=32768  # max buffered events (default: 16384)
```

Verify current values:

```bash
inotify_tuning.sh --status
# Shows current vs target for each limit, active instance count, top consumers
```

### If Code-Server Shows "Unable to Watch for File Changes"

```bash
# 1. Verify limits are set:
sysctl fs.inotify.max_user_watches
# Expected: 1048576

# 2. Check total usage across all containers:
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

# 3. If limits are set but Code-Server still shows the error:
docker restart Code-Server
# Running containers inherit limits at launch. Restart picks up the new values.

# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
inotify_tuning.sh --log
```

---

## php_fpm_max_children.sh

### What It Sets

```bash
PHP_MAX_CHILDREN=250             # target pm.max_children (default: 4-8 on unRAID)
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
```

250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
The default of 4–8 saturates immediately under load on a busy server.

### Verify

```bash
php_fpm_max_children.sh --status
# Shows current value vs target, PHP-FPM worker count

# Manual verify:
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
# Expected: pm.max_children = 250
```

### If WebGUI Is Slow Despite the Setting

```bash
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
# Look at the webgui_restart.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_restart.sh is regularly hitting step 2, the
# pm.max_children value may still be too low, or there's a PHP worker leak.

# Check running worker count:
pgrep -fc php-fpm
# Compare to pm.max_children — if equal, workers are saturated

# Increase if needed:
# master.conf: PHP_MAX_CHILDREN=350
# Then: php_fpm_max_children.sh --log  (will update and restart php-fpm)
```

---

## docker_syslog_filter.sh

### What It Creates

```
/etc/rsyslog.d/ignore-docker-veth.conf:
  if ($msg contains "veth" or $msg contains "docker0") then {
      stop
  }
```

This drops any syslog message containing "veth" or "docker0" before it reaches
any output target, including the log file. The drop rule is applied at rsyslog
level — not at the log viewer level.

### Verify

```bash
docker_syslog_filter.sh --status
# Shows filter file content and rsyslog process state

# Manual verify:
cat /etc/rsyslog.d/ignore-docker-veth.conf
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"

# Test the filter is active (should produce no syslog output):
logger "test veth message"
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
```

### If Syslog Still Has Veth Noise

```bash
# 1. Verify filter file exists with correct content:
docker_syslog_filter.sh --status

# 2. If content differs — re-apply:
docker_syslog_filter.sh --log

# 3. Verify rsyslog is using the conf.d directory:
grep -r "IncludeConfig" /etc/rsyslog.conf
# Expected: IncludeConfig /etc/rsyslog.d/*.conf  (or similar)
```

---

## clear_logs.sh

### Thresholds

```bash
LOG_MIN_SIZE_MB=10       # skip system log if under this — keep recent history
LOG_DOCKER_MAX_MB=100    # clear Docker container log only if over this

LOG_FILES=(
    "/var/log/syslog"
    "/var/log/messages"
    "/var/log/dmesg"
)
```

### Why Truncation Not Deletion

unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
descriptor open and valid while emptying content — syslogd continues writing to
the same fd without interruption. Deleting the file would orphan the file
descriptor and syslog would stop writing until restarted.

### Identifying Large Docker Logs

```bash
clear_logs.sh --status
# Shows top 10 Docker logs by size, current size vs threshold

# Find the biggest log manually:
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
```

---

## mover_stop.sh

### Stop Sequence

```
1. Check if mover is running (pgrep "emhttp.*Mover") → exit cleanly if not
2. Wall message to all logged-in terminal users
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
4. SIGTERM — mover finishes its current file operation, then stops
5. Wait 5 seconds → verify stopped
6. SIGKILL if still running — forced stop, partial files possible
7. Final verify — error if still running after SIGKILL
```

SIGTERM first because the mover can finish the file it is currently moving,
leaving no partial copies split across cache and array. SIGKILL is a last resort.

### Configuration

```bash
MOVER_STOP_TIMEOUT=30   # seconds between wall warning and SIGTERM
```

### Usage

```bash
mover_stop.sh              # check and stop if running
mover_stop.sh --status     # show current mover state and PID
mover_stop.sh --dry-run    # show what would happen without stopping
```

---

## rsync_stop.sh

### Auto-Detection Logic

rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.

**Default behavior (orchestrator detected):**
Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.

**Default behavior (no orchestrator):**
Kills rsync directly (standalone rsync.sh run).

**--full-stop:**
Kills the orchestrator first, then kills rsync. Nothing continues after this.
Use when everything needs to stop immediately.

### Container Recovery

After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
for any that were stopped by the interrupted rsync session and restarts them.
Remote containers are left for docker_watchdog.sh to recover.

Skip container recovery with `--rsync-only` — used when called by other scripts
that handle recovery themselves.

### Usage

```bash
rsync_stop.sh                        # smart stop (auto-detect orchestrator)
rsync_stop.sh --full-stop            # kill orchestrator + rsync
rsync_stop.sh --rsync-only           # kill rsync, skip container recovery
rsync_stop.sh --status               # show local and remote rsync state
rsync_stop.sh --dry-run              # preview without changes
rsync_stop.sh --full-stop --dry-run  # preview full stop
```

---

## user_scripts_stop.sh

### Process Identification

Scans `/proc/*/cmdline` for any process whose command line contains
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
what the script is named or how it was launched.

Script names are extracted from the path for display: you see which scripts are
being stopped, not just PIDs.

### Self-Exclusion

If this script is run via the User Scripts plugin, it would find its own PID in
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
from the kill list.

### Usage

```bash
user_scripts_stop.sh              # stop all User Script processes
user_scripts_stop.sh --status     # show running scripts with names and elapsed time
user_scripts_stop.sh --dry-run    # show what would be stopped
```

---

## server_reboot.sh

### Full Shutdown Sequence

```
1. Pre-flight checks (warn only — do not block):
   - rsync running?     → warn, suggest rsync_stop.sh first
   - mover running?     → warn, suggest mover_stop.sh first
   - Emby sessions?     → warn (active streams will be interrupted)

2. Wall message to all logged-in terminal users

3. unRAID dashboard notification

4. Wait REBOOT_SLEEP seconds (default: 30)

5. Graceful VM shutdown:
   virsh shutdown <each VM>  (ACPI signal — clean shutdown)
   Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond

6. /etc/rc.d/rc.libvirt stop   (VM Manager)

7. /etc/rc.d/rc.docker stop    (all containers stop)

8. sync   (flush filesystem buffers to disk)

9. /sbin/reboot
```

### Configuration

```bash
REBOOT_SLEEP=30      # seconds between warning and shutdown sequence
REBOOT_VM_WAIT=30    # seconds to wait for VMs to shut down gracefully
```

### Recommended Pre-Reboot Sequence

For a clean reboot when services are active:

```bash
rsync_stop.sh          # stop any active rsync (smart mode)
mover_stop.sh          # stop mover gracefully
server_reboot.sh --status  # check what's still running
server_reboot.sh --reason="planned maintenance"
```

### Usage

```bash
server_reboot.sh                         # reboot with 30s warning
server_reboot.sh --dry-run               # walk through without rebooting
server_reboot.sh --status                # show what would be affected
server_reboot.sh --reason="disk work"    # include reason in notification
```

---

## Full Configuration Reference

```bash
# master.conf

# ── System Watchdog ────────────────────────────────────────────────────────────
SYS_WATCHDOG_INTERVAL=300             # seconds between check cycles
SYS_WATCHDOG_STRIKES=3                # consecutive failures before reboot
SYS_WATCHDOG_REBOOT_WINDOW_HRS=2      # rate limit window
SYS_WATCHDOG_MAX_REBOOTS=3            # max reboots in window
SYS_WATCHDOG_OOM_LIMIT=3              # OOM kills/cycle for URGENT bypass

MEM_WARN_GB=10                        # warn + notify
MEM_SHUTDOWN_GB=6                     # stop non-essential containers
MEM_GB=4                              # strike → reboot
MEM_RECOVER_GB=30                     # recovery threshold

SYS_WATCHDOG_ROOTFS_CRITICAL_PCT=99   # Tier 1 trigger
SYS_WATCHDOG_FD_CRITICAL_PCT=95       # Tier 1 trigger
SYS_WATCHDOG_LOAD_MULTIPLIER=4        # Tier 3 — ×cpu_count
SYS_WATCHDOG_CPU_TEMP=85              # Tier 3 — Celsius
SYS_WATCHDOG_ZOMBIES=20               # Tier 3 — process count
SYS_WATCHDOG_VAR_LOG_PCT=80           # Tier 3 — percent full
SYS_WATCHDOG_TMP_PCT=85               # Tier 3 — percent full

SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true
SYS_WATCHDOG_ABORT_ON_PARITY=true
SYS_WATCHDOG_ABORT_ON_MOVER=true

SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=(
    "NginxProxyManager" "Authelia" "Mariadb" "Redis" "Emby" "Dispatcharr"
)
SYS_WATCHDOG_REQUIRED_CONTAINERS=()  # containers that must be running

# State files:
SYS_WATCHDOG_STATE_FILE="/tmp/sys_watchdog_state.db"
SYS_WATCHDOG_REBOOT_LOG="/tmp/sys_watchdog_reboots.db"
SYS_WATCHDOG_FAILED_FILE="/tmp/sys_watchdog_failed.db"
SYS_WATCHDOG_OOM_FILE="/tmp/sys_watchdog_oom.db"

# ── Resource Watchdog ──────────────────────────────────────────────────────────
RW_ENABLED=true
RW_STATE_FILE="/tmp/resource_watchdog_state.db"

RW_RAM_SOFT_GB=20
RW_RAM_MEDIUM_GB=15
RW_RAM_HARD_GB=10
RW_RAM_RECOVER_GB=25

RW_LOAD_SOFT_MULTIPLIER=2.0
RW_LOAD_MEDIUM_MULTIPLIER=3.0
RW_RECOVER_CYCLES=3

RW_SABNZBD_ENABLED=true
RW_SABNZBD_SPEED_SOFT="50M"
RW_SABNZBD_SPEED_MEDIUM="10M"
RW_QBIT_ENABLED=true
RW_QBIT_DL_SOFT=51200        # KB/s
RW_QBIT_DL_MEDIUM=10240

RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis")

# ── Per-Host (host*.conf) ───────────────────────────────────────────────
HOST1_RW_PAUSE_CONTAINERS=("Tdarr" "HandBrake")
HOST1_RW_STOP_CONTAINERS=("LocalAI" "Satisfactory")
HOST1_SABNZBD_URL="http://localhost:8080"
HOST1_SABNZBD_API_KEY="your-api-key"
HOST1_QBIT_URL="http://localhost:8090"
HOST1_QBIT_USERNAME="admin"
HOST1_QBIT_PASSWORD="your-password"

# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30

# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768

# ── PHP-FPM ────────────────────────────────────────────────────────────────────
PHP_MAX_CHILDREN=250
PHP_CONF="/etc/php83/php-fpm.d/www.conf"

# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"

# ── Log Cleaner ────────────────────────────────────────────────────────────────
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100

# ── Mover Stop ─────────────────────────────────────────────────────────────────
MOVER_STOP_TIMEOUT=30

# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30
```

---

## Troubleshooting

### system_watchdog.sh Not Starting

```bash
# Check if already running (acquire_lock prevents second instance):
pgrep -a -f system_watchdog.sh

# Check state file permissions:
ls -la /tmp/sys_watchdog_*.db

# Run with --log to see startup:
system_watchdog.sh --log
```

### system_watchdog Rebooted Unexpectedly

```bash
# Check reboot log:
cat /tmp/sys_watchdog_reboots.db
# Shows timestamp and reason for each watchdog-triggered reboot

# Check what condition triggered it:
# Look in /var/log/syslog for "system_watchdog" near the reboot time
grep "system_watchdog" /var/log/syslog | tail -20
```

### resource_watchdog Paused Containers It Shouldn't Have

```bash
# Check current state:
resource_watchdog.sh --status

# Add the container to RW_CRITICAL_CONTAINERS in master.conf:
RW_CRITICAL_CONTAINERS=("Emby" "NginxProxyManager" "Authelia" "Mariadb" "Redis" "MyContainer")

# Un-pause manually if needed:
docker unpause MyContainer
```

### rsync_stop Killed the Wrong Thing

If --full-stop killed an orchestrator you didn't intend to kill:

```bash
# Next time use default mode (no --full-stop) to kill only rsync subprocess.
# To verify what would be killed before running:
rsync_stop.sh --status       # shows running rsync and detected orchestrators
rsync_stop.sh --dry-run      # shows smart mode decision
rsync_stop.sh --full-stop --dry-run  # shows full-stop decision
```

### WebGUI Recovery After All Three Steps Failed

```bash
# Check if processes are running:
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
pgrep emhttpd   && echo "emhttp: yes" || echo "emhttp: no"
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"

# Check recent nginx errors:
cat /var/log/nginx/error.log | tail -20

# Check emhttp log:
tail -20 /var/log/syslog | grep emhttp

# Last resort — reboot:
server_reboot.sh --reason="WebGUI unrecoverable"
```

### PHP-FPM Config Not Found After unRAID Update

unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
errors with "config file not found":

```bash
# Find the new config path:
find /etc -name "www.conf" 2>/dev/null

# Update PHP_CONF in master.conf:
PHP_CONF="/etc/php84/php-fpm.d/www.conf"   # example for php84

# Verify with:
php_fpm_max_children.sh --status
```