Varaverk/System_Essentials/Manual-System_Essentials.md

# ━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
adding or reordering scripts at array start.

> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) and their configuration live in
> `Plugin/unraid/System_Essentials/`. Watchdog configuration (`stability_watchdog.sh`,
> `resource_watchdog.sh`, `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives
> in `Watchdogs/Manual-Watchdogs.md`.

---

## ━━━ CONTENTS ━━━

- [ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order](#array_start_scripts--array_stop_scripts-order)
- [conf_sync.sh](#conf_syncsh)
- [conf_cache_save.sh](#conf_cache_savesh)
- [conf_cache_restore.sh](#conf_cache_restoresh)
- [inotify_tuning.sh](#inotify_tuningsh)
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
- [clear_logs.sh](#clear_logssh)
- [rsync_stop.sh](#rsync_stopsh)
- [server_reboot.sh](#server_rebootsh)
- [Full Configuration Reference](#full-configuration-reference)
- [Troubleshooting](#troubleshooting)

---

## Output Tiers

All scripts use a two-tier output model: `echo` lines are always visible; `log`
lines only appear when `--log` is passed.

**One-shot scripts** (`conf_sync.sh`, `conf_cache_save.sh`, `conf_cache_restore.sh`,
`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`, `rsync_stop.sh`,
`server_reboot.sh`): without `--log`, section headers, per-step results, and the
final summary are visible. Per-item detail suppressed.

---

## ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order

> **Order matters.** Scripts that set up conditions other scripts depend on must run
> first. Getting the order wrong causes subtle failures.

```bash
# master.conf (current order — from ARRAY_START_SCRIPTS)
ARRAY_START_SCRIPTS=(
    "Plugin/unraid/System_Essentials/unraid_api_key_renew.sh"   # plugin API key — ephemeral registry
    "System_Essentials/conf_sync.sh"              # 1st cross-server op — pull partner confs into RAM cache
    "System_Essentials/conf_cache_restore.sh"     # fill gaps if partner was down at boot
    "Transcodes/ramdisk_setup.sh"                 # ramdisk before Emby starts
    "System_Essentials/docker_syslog_filter.sh"   # before veth interfaces are created
    "Plugin/unraid/System_Essentials/php_fpm_max_children.sh"   # WebGUI tuning
    "System_Essentials/inotify_tuning.sh"         # before docker_network_connect and continuous scripts
    "Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
    "Arrs_Stack/start_webhook_listener.sh"             # arr upgrade webhook — continuous
    "Fallback/fallback.sh"                        # mutual failover — continuous
)

ARRAY_STOP_SCRIPTS=(
    "System_Essentials/conf_cache_save.sh"          # FIRST: snapshot RAM cache while fresh
    "Plugin/unraid/System_Essentials/user_scripts_stop.sh"
    "Fallback/fallback.sh --stop"
    "System_Essentials/rsync_stop.sh --rsync-only"
    "Plugin/unraid/System_Essentials/mover_stop.sh"
    "Docker_Essentials/docker_container_stop.sh"
)
```

Why conf_sync before conf_cache_restore: the sync runs first to get the freshest
possible partner conf. The restore only fills in for confs that the sync couldn't
fetch — it is a fallback, not the primary path.

Why docker_syslog_filter before inotify: the filter must be in place before any
container creates veth interfaces. inotify must be set before docker_network_connect
and the continuous scripts (start_webhook_listener, fallback), which are the first
steps that actually touch containers.

Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
`/tmp/varaverk/conf/` is wiped on reboot. Saving it must happen before anything
else shuts down or changes state.

---

## conf_sync.sh

### Modes

```
conf_sync.sh                   Full sync: pull from all partners + push to all partners
conf_sync.sh --push-only       Push own conf to all partners (fast — for conf-save hook)
conf_sync.sh --pull-only       Pull partner confs only (intermediate orchestrator)
conf_sync.sh --dry-run         Show what would happen, no changes
conf_sync.sh --log             Verbose output
```

### What It Syncs

- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
  via SCP → writes to local `/tmp/varaverk/conf/${partner_id}.conf`
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
  `/tmp/varaverk/conf/${my_id}.conf` via SCP
- **Own conf in local cache**: copies own conf to `/tmp/varaverk/conf/${my_id}.conf`
  on full sync (so the cache has a complete snapshot of all confs)

Only partner confs are sourced from cache — `load_config.sh` always reads own conf
from disk to avoid sourcing a stale cached copy.

### Remote SCRIPTS_DIR Resolution

The pull path reads the partner's `/boot/config/plugins/varaverk/varaverk.cfg` to
find their actual `SCRIPTS_DIR` before building the SCP path. This handles the case
where the partner is in appdata storage mode and their conf is at
`/mnt/user/appdata/Varaverk/Configurations/` rather than the internal path.

### PARTNERSHIP_ENABLED Gate

conf_sync.sh calls `require_partnership` — if `PARTNERSHIP_ENABLED=false`, it exits
silently with 0. The conf cache will be empty for partner confs while partnership
is disabled.

### If Conf Pull Fails at Boot

If the partner is unreachable, `conf_sync.sh` logs a warning and exits 1.
`conf_cache_restore.sh` then runs (next in ARRAY_START_SCRIPTS) and loads the
persistent backup from `$PERSISTENT_CONF_CACHE` if available.

A notification fires if any partner fails — check partner reachability via Tailscale.

---

## conf_cache_save.sh

### What It Does

At array stop, copies all partner confs from `/tmp/varaverk/conf/` to
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
can't reach the partner.

`conf_cache_watchdog.sh` (in `Watchdogs/System/`) refreshes this backup every
15 minutes while the partner is offline — keeping it current even during extended
outages.

### PARTNERSHIP_ENABLED Gate

Exits silently when `PARTNERSHIP_ENABLED=false`. No backup is written.

### Usage

```bash
conf_cache_save.sh              # save partner confs from RAM cache (called at array stop)
conf_cache_save.sh --dry-run    # show what would be saved
conf_cache_save.sh --log        # verbose output
```

---

## conf_cache_restore.sh

### What It Does

At array start (after `conf_sync.sh`), checks which partner confs are missing from
`/tmp/varaverk/conf/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
if a backup exists there.

Always removes the persistent backup when done — whether used or not. On a normal
reboot where the partner was reachable, the sync already populated the cache and the
backup is cleaned up without being used.

### PARTNERSHIP_ENABLED Gate

Exits silently when `PARTNERSHIP_ENABLED=false`.

### Usage

```bash
conf_cache_restore.sh              # restore missing confs from backup (called at array start)
conf_cache_restore.sh --dry-run    # show what would be restored
conf_cache_restore.sh --log        # verbose output
```

---

## inotify_tuning.sh

### What It Sets

```bash
INOTIFY_MAX_INSTANCES=1024       # max inotify fd objects per user (default: 128)
INOTIFY_MAX_WATCHES=1048576      # max watches shared across all users (default: 8192)
INOTIFY_MAX_QUEUED_EVENTS=32768  # max buffered events (default: 16384)
```

Verify current values:

```bash
inotify_tuning.sh --status
# Shows current vs target for each limit, active instance count, top consumers
```

### If Code-Server Shows "Unable to Watch for File Changes"

```bash
# 1. Verify limits are set:
sysctl fs.inotify.max_user_watches
# Expected: 1048576

# 2. Check total usage across all containers:
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

# 3. If limits are set but Code-Server still shows the error:
docker restart Code-Server
# Running containers inherit limits at launch. Restart picks up the new values.

# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
inotify_tuning.sh --log
```

---

## docker_syslog_filter.sh

### What It Creates

```
/etc/rsyslog.d/ignore-docker-veth.conf:
  if ($msg contains "veth" or $msg contains "docker0") then {
      stop
  }
```

This drops any syslog message containing "veth" or "docker0" before it reaches
any output target, including the log file.

### Verify

```bash
docker_syslog_filter.sh --status
# Shows filter file content and rsyslog process state

# Manual verify:
cat /etc/rsyslog.d/ignore-docker-veth.conf
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"

# Test the filter is active (should produce no syslog output):
logger "test veth message"
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"
```

### If Syslog Still Has Veth Noise

```bash
# 1. Verify filter file exists with correct content:
docker_syslog_filter.sh --status

# 2. If content differs — re-apply:
docker_syslog_filter.sh --log

# 3. Verify rsyslog is using the conf.d directory:
grep -r "IncludeConfig" /etc/rsyslog.conf
# Expected: IncludeConfig /etc/rsyslog.d/*.conf  (or similar)
```

---

## clear_logs.sh

### Thresholds

```bash
LOG_MIN_SIZE_MB=10       # skip system log if under this — keep recent history
LOG_DOCKER_MAX_MB=100    # clear Docker container log only if over this

LOG_FILES=(
    "/var/log/syslog"
    "/var/log/messages"
    "/var/log/dmesg"
)
```

### Why Truncation Not Deletion

unRAID writes logs to tmpfs (`/var/log`). Truncation (`: > file`) keeps the file
descriptor open and valid while emptying content — syslogd continues writing to
the same fd without interruption. Deleting the file would orphan the file
descriptor and syslog would stop writing until restarted.

### Identifying Large Docker Logs

```bash
clear_logs.sh --status
# Shows top 10 Docker logs by size, current size vs threshold

# Find the biggest log manually:
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
```

---

## rsync_stop.sh

### Auto-Detection Logic

rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync)
is the parent of the running rsync process by scanning lock files in `$LOCK_DIR`.

**Default behavior (orchestrator detected):**
Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the
next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.

**Default behavior (no orchestrator):**
Kills rsync directly (standalone rsync.sh run).

**--full-stop:**
Kills the orchestrator first, then kills rsync. Nothing continues after this.
Use when everything needs to stop immediately.

### Container Recovery

After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTAINER_NAMES`
for any that were stopped by the interrupted rsync session and restarts them.
Remote containers are left for docker_watchdog.sh to recover.

Skip container recovery with `--rsync-only` — used when called by array_stopping.sh
(which handles container stop separately via docker_container_stop.sh).

### Usage

```bash
rsync_stop.sh                        # smart stop (auto-detect orchestrator)
rsync_stop.sh --full-stop            # kill orchestrator + rsync
rsync_stop.sh --rsync-only           # kill rsync, skip container recovery
rsync_stop.sh --status               # show local and remote rsync state
rsync_stop.sh --dry-run              # preview without changes
rsync_stop.sh --full-stop --dry-run  # preview full stop
```

---

## server_reboot.sh

### Full Shutdown Sequence

```
1. Pre-flight checks (warn only — do not block):
   - rsync running?     → warn, suggest rsync_stop.sh first
   - mover running?     → warn, suggest mover_stop.sh first
   - Emby sessions?     → warn (active streams will be interrupted)

2. Wall message to all logged-in terminal users

3. unRAID dashboard notification

4. Wait REBOOT_SLEEP seconds (default: 30)

5. Graceful VM shutdown:
   virsh shutdown <each VM>  (ACPI signal — clean shutdown)
   Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond

6. /etc/rc.d/rc.libvirt stop   (VM Manager)

7. Orchestrators/array_stopping.sh — safe ordered array stop:
   - conf_cache_save.sh           snapshot partner conf RAM cache
   - user_scripts_stop.sh         stop background User Scripts
   - fallback.sh --stop           graceful fallback teardown
   - rsync_stop.sh --rsync-only   kill active rsync transfers
   - mover_stop.sh                stop mover
   - docker_container_stop.sh     stop all containers gracefully

8. sync   (flush filesystem buffers to disk)

9. /sbin/reboot
```

### Configuration

```bash
REBOOT_SLEEP=30      # seconds between warning and shutdown sequence
REBOOT_VM_WAIT=30    # seconds to wait for VMs to shut down gracefully
```

### Recommended Pre-Reboot Sequence

For a clean reboot when services are active:

```bash
rsync_stop.sh          # stop any active rsync (smart mode)
server_reboot.sh --status  # check what's still running
server_reboot.sh --reason="planned maintenance"
```

### Usage

```bash
server_reboot.sh                         # reboot with 30s warning
server_reboot.sh --dry-run               # walk through without rebooting
server_reboot.sh --status                # show what would be affected
server_reboot.sh --reason="disk work"    # include reason in notification
```

---

## Full Configuration Reference

```bash
# master.conf

# ── Conf Sync ──────────────────────────────────────────────────────────────────
CONF_SYNC_ENABLED=true          # toggle: false disables conf_sync.sh entirely

# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768
INOTIFY_WARN_PCT=80             # warn (in reports) if instances exceed this % of limit

# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"

# ── Log Cleaner ────────────────────────────────────────────────────────────────
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100

# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30

# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
STATE_DIR="${DATA_DIR}/state"
PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
```

---

## Troubleshooting

### Partner Conf Not In RAM Cache After Boot

```bash
# Check what's in the RAM cache:
ls -la /tmp/varaverk/conf/

# Check what's in the persistent backup:
ls -la "$PERSISTENT_CONF_CACHE/"    # set SCRIPTS_DIR first or use full path

# Re-run conf sync manually:
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log

# If partner is reachable but pull failed, check SSH key:
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"
```

### Persistent Backup Is Stale or Empty

```bash
# conf_cache_watchdog.sh refreshes the backup while partner is offline.
# Check watchdog state:
cat "$STATE_DIR/conf_cache_watchdog_state.db"

# Force a manual backup from current RAM cache:
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log
```

### rsync_stop Killed the Wrong Thing

If --full-stop killed an orchestrator you didn't intend to kill:

```bash
# Next time use default mode (no --full-stop) to kill only rsync subprocess.
# To verify what would be killed before running:
rsync_stop.sh --status       # shows running rsync and detected orchestrators
rsync_stop.sh --dry-run      # shows smart mode decision
rsync_stop.sh --full-stop --dry-run  # shows full-stop decision
```

### inotify Exhaustion After Boot

```bash
# Verify limits are applied:
sysctl fs.inotify.max_user_watches    # expect 1048576
sysctl fs.inotify.max_user_instances  # expect 1024

# If not set — run manually:
inotify_tuning.sh --log

# Check current usage:
inotify_tuning.sh --status
```

### Syslog Still Noisy After Array Start

```bash
# Check filter is in place:
docker_syslog_filter.sh --status

# Re-apply if needed:
docker_syslog_filter.sh --log

# Restart rsyslog to pick up the filter:
/etc/rc.d/rc.rsyslogd restart
```