Varaverk/System_Essentials/Manual-System_Essentials.md

# ━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.

> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
> procedures are in `Watchdogs/Manual-Watchdogs.md`.

---

## ━━━ CONTENTS ━━━

- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
- [webgui_watchdog.sh](#webgui_watchdogsh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
- [clear_logs.sh](#clear_logssh)
- [mover_stop.sh](#mover_stopsh)
- [rsync_stop.sh](#rsync_stopsh)
- [user_scripts_stop.sh](#user_scripts_stopsh)
- [server_reboot.sh](#server_rebootsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)

---

## Output Tiers

All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.

**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
Per-check detail suppressed.

**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.

---

## ARRAY_START_SCRIPTS Order

> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
> scripts.** Getting it wrong causes subtle failures that don't show up
> immediately.

```bash
# master.conf
ARRAY_START_SCRIPTS=(
    "inotify_tuning.sh"           # 1 — FIRST: kernel limits must be set before
                                   #    any container starts. Containers inherit
                                   #    inotify limits at launch, not dynamically.
    "docker_syslog_filter.sh"     # 2 — SECOND: before any veth interfaces are
                                   #    created. If a container starts first, its
                                   #    veth creation is already in syslog.
    "php_fpm_max_children.sh"     # 3 — before WebGUI is under load
    "ramdisk_setup.sh"            # (from Transcodes/) before Emby starts
    ...
)
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
```

Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.

Why docker_syslog_filter SECOND: The filter must be in place before any container
starts creating veth interfaces. The first container start after array start
generates veth messages — these will appear in syslog if the filter isn't active.

---

## webgui_watchdog.sh

### Escalation Logic

```
curl $WEBGUI_URL → 200 OK → exit 0 (silent)

Not responding:
  1. platform_restart_service nginx
     wait WEBGUI_NGINX_WAIT (15s) → recheck
     → recovered: notify, exit 0

  2. platform_restart_service php-fpm
     wait WEBGUI_PHP_WAIT (10s) → recheck
     → recovered: notify, exit 0

  3. platform_restart_service emhttp
     wait WEBGUI_EMHTTP_WAIT (30s) → recheck
     → recovered: notify, exit 0

  All three failed → notify warning, exit 1
```

### Configuration

```bash
WEBGUI_URL="http://localhost"   # URL to check
WEBGUI_TIMEOUT=5                # curl timeout in seconds
WEBGUI_NGINX_WAIT=15            # seconds after nginx restart before recheck
WEBGUI_PHP_WAIT=10              # seconds after php-fpm restart before recheck
WEBGUI_EMHTTP_WAIT=30           # seconds after emhttp restart before recheck
```

### WebGUI Frozen — Manual Recovery

```bash
# Check which services are running:
webgui_watchdog.sh --status

# Try manual restart sequence (mirrors what the script calls):
# Source the ecosystem first to get platform functions:
source /boot/config/plugins/varaverk/load_config.sh
platform_restart_service nginx
# wait 15s, then:
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"

# If nginx did not fix it, php-fpm:
platform_restart_service php-fpm

# If still down, emhttp:
platform_restart_service emhttp

# Raw equivalents (no source needed — paste directly into terminal):
# /etc/rc.d/rc.nginx restart
# /etc/rc.d/rc.php-fpm restart
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start

# If all three failed:
server_reboot.sh --status   # check for active sessions first
```

---

## inotify_tuning.sh

### What It Sets

```bash
INOTIFY_MAX_INSTANCES=1024       # max inotify fd objects per user (default: 128)
INOTIFY_MAX_WATCHES=1048576      # max watches shared across all users (default: 8192)
INOTIFY_MAX_QUEUED_EVENTS=32768  # max buffered events (default: 16384)
```

Verify current values:

```bash
inotify_tuning.sh --status
# Shows current vs target for each limit, active instance count, top consumers
```

### If Code-Server Shows "Unable to Watch for File Changes"

```bash
# 1. Verify limits are set:
sysctl fs.inotify.max_user_watches
# Expected: 1048576

# 2. Check total usage across all containers:
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

# 3. If limits are set but Code-Server still shows the error:
docker restart Code-Server
# Running containers inherit limits at launch. Restart picks up the new values.

# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
inotify_tuning.sh --log
```

---

## php_fpm_max_children.sh

### What It Sets

```bash
PHP_MAX_CHILDREN=250             # target pm.max_children (default: 4-8 on unRAID)
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
```

250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
The default of 4–8 saturates immediately under load on a busy server.

### Verify

```bash
php_fpm_max_children.sh --status
# Shows current value vs target, PHP-FPM worker count

# Manual verify:
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
# Expected: pm.max_children = 250
```

### If WebGUI Is Slow Despite the Setting

```bash
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
# pm.max_children value may still be too low, or there's a PHP worker leak.

# Check running worker count:
pgrep -fc php-fpm
# Compare to pm.max_children — if equal, workers are saturated

# Increase if needed:
# master.conf: PHP_MAX_CHILDREN=350
# Then: php_fpm_max_children.sh --log  (will update and restart php-fpm)
```

---

## docker_syslog_filter.sh

### What It Creates

```
/etc/rsyslog.d/ignore-docker-veth.conf:
  if ($msg contains "veth" or $msg contains "docker0") then {
      stop
  }
```

This drops any syslog message containing "veth" or "docker0" before it reaches
any output target, including the log file. The drop rule is applied at rsyslog
level — not at the log viewer level.

### Verify

```bash
docker_syslog_filter.sh --status
# Shows filter file content and rsyslog process state

# Manual verify:
cat /etc/rsyslog.d/ignore-docker-veth.conf
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"

# Test the filter is active (should produce no syslog output):
logger "test veth message"
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
```

### If Syslog Still Has Veth Noise

```bash
# 1. Verify filter file exists with correct content:
docker_syslog_filter.sh --status

# 2. If content differs — re-apply:
docker_syslog_filter.sh --log

# 3. Verify rsyslog is using the conf.d directory:
grep -r "IncludeConfig" /etc/rsyslog.conf
# Expected: IncludeConfig /etc/rsyslog.d/*.conf  (or similar)
```

---

## clear_logs.sh

### Thresholds

```bash
LOG_MIN_SIZE_MB=10       # skip system log if under this — keep recent history
LOG_DOCKER_MAX_MB=100    # clear Docker container log only if over this

LOG_FILES=(
    "/var/log/syslog"
    "/var/log/messages"
    "/var/log/dmesg"
)
```

### Why Truncation Not Deletion

unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
descriptor open and valid while emptying content — syslogd continues writing to
the same fd without interruption. Deleting the file would orphan the file
descriptor and syslog would stop writing until restarted.

### Identifying Large Docker Logs

```bash
clear_logs.sh --status
# Shows top 10 Docker logs by size, current size vs threshold

# Find the biggest log manually:
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
```

---

## mover_stop.sh

### Stop Sequence

```
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
2. Wall message to all logged-in terminal users
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
4. SIGTERM — mover finishes its current file operation, then stops
5. Wait 5 seconds → verify stopped
6. SIGKILL if still running — forced stop, partial files possible
7. Final verify — error if still running after SIGKILL
```

SIGTERM first because the mover can finish the file it is currently moving,
leaving no partial copies split across cache and array. SIGKILL is a last resort.

### Configuration

```bash
MOVER_STOP_TIMEOUT=30   # seconds between wall warning and SIGTERM
```

### Usage

```bash
mover_stop.sh              # check and stop if running
mover_stop.sh --status     # show current mover state and PID
mover_stop.sh --dry-run    # show what would happen without stopping
```

---

## rsync_stop.sh

### Auto-Detection Logic

rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.

**Default behavior (orchestrator detected):**
Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.

**Default behavior (no orchestrator):**
Kills rsync directly (standalone rsync.sh run).

**--full-stop:**
Kills the orchestrator first, then kills rsync. Nothing continues after this.
Use when everything needs to stop immediately.

### Container Recovery

After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
for any that were stopped by the interrupted rsync session and restarts them.
Remote containers are left for docker_watchdog.sh to recover.

Skip container recovery with `--rsync-only` — used when called by other scripts
that handle recovery themselves.

### Usage

```bash
rsync_stop.sh                        # smart stop (auto-detect orchestrator)
rsync_stop.sh --full-stop            # kill orchestrator + rsync
rsync_stop.sh --rsync-only           # kill rsync, skip container recovery
rsync_stop.sh --status               # show local and remote rsync state
rsync_stop.sh --dry-run              # preview without changes
rsync_stop.sh --full-stop --dry-run  # preview full stop
```

---

## user_scripts_stop.sh

### Process Identification

Scans `/proc/*/cmdline` for any process whose command line contains
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
what the script is named or how it was launched.

Script names are extracted from the path for display: you see which scripts are
being stopped, not just PIDs.

### Self-Exclusion

If this script is run via the User Scripts plugin, it would find its own PID in
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
from the kill list.

### Usage

```bash
user_scripts_stop.sh              # stop all User Script processes
user_scripts_stop.sh --status     # show running scripts with names and elapsed time
user_scripts_stop.sh --dry-run    # show what would be stopped
```

---

## server_reboot.sh

### Full Shutdown Sequence

```
1. Pre-flight checks (warn only — do not block):
   - rsync running?     → warn, suggest rsync_stop.sh first
   - mover running?     → warn, suggest mover_stop.sh first
   - Emby sessions?     → warn (active streams will be interrupted)

2. Wall message to all logged-in terminal users

3. unRAID dashboard notification

4. Wait REBOOT_SLEEP seconds (default: 30)

5. Graceful VM shutdown:
   virsh shutdown <each VM>  (ACPI signal — clean shutdown)
   Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond

6. /etc/rc.d/rc.libvirt stop   (VM Manager)

7. Orchestrators/array_stopping.sh — safe ordered array stop:
   - user_scripts_stop.sh        stop background User Scripts
   - fallback.sh --stop          graceful fallback teardown
   - rsync_stop.sh --rsync-only  kill active rsync transfers
   - mover_stop.sh               stop mover
   - docker_container_stop.sh    stop all containers gracefully

8. sync   (flush filesystem buffers to disk)

9. /sbin/reboot
```

### Configuration

```bash
REBOOT_SLEEP=30      # seconds between warning and shutdown sequence
REBOOT_VM_WAIT=30    # seconds to wait for VMs to shut down gracefully
```

### Recommended Pre-Reboot Sequence

For a clean reboot when services are active:

```bash
rsync_stop.sh          # stop any active rsync (smart mode)
mover_stop.sh          # stop mover gracefully
server_reboot.sh --status  # check what's still running
server_reboot.sh --reason="planned maintenance"
```

### Usage

```bash
server_reboot.sh                         # reboot with 30s warning
server_reboot.sh --dry-run               # walk through without rebooting
server_reboot.sh --status                # show what would be affected
server_reboot.sh --reason="disk work"    # include reason in notification
```

---

## Full Configuration Reference

> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.

```bash
# master.conf

# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
WEBGUI_URL="http://localhost"
WEBGUI_TIMEOUT=5
WEBGUI_NGINX_WAIT=15
WEBGUI_PHP_WAIT=10
WEBGUI_EMHTTP_WAIT=30

# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768

# ── PHP-FPM ────────────────────────────────────────────────────────────────────
PHP_MAX_CHILDREN=250
PHP_CONF="/etc/php83/php-fpm.d/www.conf"

# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"

# ── Log Cleaner ────────────────────────────────────────────────────────────────
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100

# ── Mover Stop ─────────────────────────────────────────────────────────────────
MOVER_STOP_TIMEOUT=30

# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30
```

---

## Troubleshooting

> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.

### rsync_stop Killed the Wrong Thing

If --full-stop killed an orchestrator you didn't intend to kill:

```bash
# Next time use default mode (no --full-stop) to kill only rsync subprocess.
# To verify what would be killed before running:
rsync_stop.sh --status       # shows running rsync and detected orchestrators
rsync_stop.sh --dry-run      # shows smart mode decision
rsync_stop.sh --full-stop --dry-run  # shows full-stop decision
```

### WebGUI Recovery After All Three Steps Failed

```bash
# Check if processes are running:
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
pgrep emhttpd   && echo "emhttp: yes" || echo "emhttp: no"
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"

# Check recent nginx errors:
cat /var/log/nginx/error.log | tail -20

# Check emhttp log:
tail -20 /var/log/syslog | grep emhttp

# Last resort — reboot:
server_reboot.sh --reason="WebGUI unrecoverable"
```

### PHP-FPM Config Not Found After unRAID Update

unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
errors with "config file not found":

```bash
# Find the new config path:
find /etc -name "www.conf" 2>/dev/null

# Update PHP_CONF in master.conf:
PHP_CONF="/etc/php84/php-fpm.d/www.conf"   # example for php84

# Verify with:
php_fpm_max_children.sh --status
```