Varaverk/System_Essentials/Manual-System_Essentials.md

━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━

Configuration reference, operational procedures, and troubleshooting for system-level scripts. Read the ARRAY_START_SCRIPTS order section before adding or reordering scripts at array start.

Platform-specific scripts (webgui_watchdog.sh, php_fpm_max_children.sh, mover_stop.sh, user_scripts_stop.sh) and their configuration live in Plugin/unraid/System_Essentials/. Watchdog configuration (stability_watchdog.sh, resource_watchdog.sh, docker_watchdog.sh, System/storage_watchdog.sh) lives in Watchdogs/Manual-Watchdogs.md.

---

━━━ CONTENTS ━━━

• ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order
• conf_sync.sh
• conf_cache_save.sh
• conf_cache_restore.sh
• inotify_tuning.sh
• docker_syslog_filter.sh
• clear_logs.sh
• rsync_stop.sh
• server_reboot.sh
• Full Configuration Reference
• Troubleshooting

---

Output Tiers

All scripts use a two-tier output model: echo lines are always visible; log lines only appear when --log is passed.

One-shot scripts (conf_sync.sh, conf_cache_save.sh, conf_cache_restore.sh, clear_logs.sh, docker_syslog_filter.sh, inotify_tuning.sh, rsync_stop.sh, server_reboot.sh): without --log, section headers, per-step results, and the final summary are visible. Per-item detail suppressed.

---

ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order

Order matters. Scripts that set up conditions other scripts depend on must run first. Getting the order wrong causes subtle failures.

# master.conf (current order — from ARRAY_START_SCRIPTS)
ARRAY_START_SCRIPTS=(
    "Plugin/unraid/System_Essentials/unraid_api_key_renew.sh"   # plugin API key — ephemeral registry
    "System_Essentials/conf_sync.sh"              # 1st cross-server op — pull partner confs into RAM cache
    "System_Essentials/conf_cache_restore.sh"     # fill gaps if partner was down at boot
    "Transcodes/ramdisk_setup.sh"                 # ramdisk before Emby starts
    "System_Essentials/docker_syslog_filter.sh"   # before veth interfaces are created
    "Plugin/unraid/System_Essentials/php_fpm_max_children.sh"   # WebGUI tuning
    "System_Essentials/inotify_tuning.sh"         # before docker_network_connect and continuous scripts
    "Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
    "Tools/claude_startup.sh"                     # persist Claude data to appdata; re-symlink on boot
    "Media/start_webhook_listener.sh"             # arr upgrade webhook — continuous
    "Fallback/fallback.sh"                        # mutual failover — continuous
)

ARRAY_STOP_SCRIPTS=(
    "System_Essentials/conf_cache_save.sh"          # FIRST: snapshot RAM cache while fresh
    "Plugin/unraid/System_Essentials/user_scripts_stop.sh"
    "Fallback/fallback.sh --stop"
    "System_Essentials/rsync_stop.sh --rsync-only"
    "Plugin/unraid/System_Essentials/mover_stop.sh"
    "Docker_Essentials/docker_container_stop.sh"
)

Why conf_sync before conf_cache_restore: the sync runs first to get the freshest possible partner conf. The restore only fills in for confs that the sync couldn't fetch — it is a fallback, not the primary path.

Why docker_syslog_filter before inotify: the filter must be in place before any container creates veth interfaces. inotify must be set before docker_network_connect and the continuous scripts (start_webhook_listener, fallback), which are the first steps that actually touch containers.

Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at /tmp/.cache/vv/d/ is wiped on reboot. Saving it must happen before anything else shuts down or changes state.

---

conf_sync.sh

Modes

conf_sync.sh                   Full sync: pull from all partners + push to all partners
conf_sync.sh --push-only       Push own conf to all partners (fast — for conf-save hook)
conf_sync.sh --pull-only       Pull partner confs only (intermediate orchestrator)
conf_sync.sh --dry-run         Show what would happen, no changes
conf_sync.sh --log             Verbose output

What It Syncs

• Pull: reads the partner's Configurations/${partner_id}.conf from their disk via SCP → writes to local /tmp/.cache/vv/d/${partner_id}.conf
• Push: sends own Configurations/${my_id}.conf to partner's /tmp/.cache/vv/d/${my_id}.conf via SCP
• Own conf in local cache: copies own conf to /tmp/.cache/vv/d/${my_id}.conf on full sync (so the cache has a complete snapshot of all confs)

Only partner confs are sourced from cache — load_config.sh always reads own conf from disk to avoid sourcing a stale cached copy.

Remote SCRIPTS_DIR Resolution

The pull path reads the partner's /boot/config/plugins/varaverk/varaverk.cfg to find their actual SCRIPTS_DIR before building the SCP path. This handles the case where the partner is in appdata storage mode and their conf is at /mnt/user/appdata/Varaverk/Configurations/ rather than the internal path.

PARTNERSHIP_ENABLED Gate

conf_sync.sh calls require_partnership — if PARTNERSHIP_ENABLED=false, it exits silently with 0. The conf cache will be empty for partner confs while partnership is disabled.

If Conf Pull Fails at Boot

If the partner is unreachable, conf_sync.sh logs a warning and exits 1. conf_cache_restore.sh then runs (next in ARRAY_START_SCRIPTS) and loads the persistent backup from $PERSISTENT_CONF_CACHE if available.

A notification fires if any partner fails — check partner reachability via Tailscale.

---

conf_cache_save.sh

What It Does

At array stop, copies all partner confs from /tmp/.cache/vv/d/ to $PERSISTENT_CONF_CACHE. Own conf is skipped (always on disk). The backup survives the reboot and is used by conf_cache_restore.sh at next array start if the sync can't reach the partner.

conf_cache_watchdog.sh (in Watchdogs/System/) refreshes this backup every 15 minutes while the partner is offline — keeping it current even during extended outages.

PARTNERSHIP_ENABLED Gate

Exits silently when PARTNERSHIP_ENABLED=false. No backup is written.

Usage

conf_cache_save.sh              # save partner confs from RAM cache (called at array stop)
conf_cache_save.sh --dry-run    # show what would be saved
conf_cache_save.sh --log        # verbose output

---

conf_cache_restore.sh

What It Does

At array start (after conf_sync.sh), checks which partner confs are missing from /tmp/.cache/vv/d/. For each missing conf, loads it from $PERSISTENT_CONF_CACHE if a backup exists there.

Always removes the persistent backup when done — whether used or not. On a normal reboot where the partner was reachable, the sync already populated the cache and the backup is cleaned up without being used.

PARTNERSHIP_ENABLED Gate

Exits silently when PARTNERSHIP_ENABLED=false.

Usage

conf_cache_restore.sh              # restore missing confs from backup (called at array start)
conf_cache_restore.sh --dry-run    # show what would be restored
conf_cache_restore.sh --log        # verbose output

---

inotify_tuning.sh

What It Sets

INOTIFY_MAX_INSTANCES=1024       # max inotify fd objects per user (default: 128)
INOTIFY_MAX_WATCHES=1048576      # max watches shared across all users (default: 8192)
INOTIFY_MAX_QUEUED_EVENTS=32768  # max buffered events (default: 16384)

Verify current values:

inotify_tuning.sh --status
# Shows current vs target for each limit, active instance count, top consumers

If Code-Server Shows "Unable to Watch for File Changes"

# 1. Verify limits are set:
sysctl fs.inotify.max_user_watches
# Expected: 1048576

# 2. Check total usage across all containers:
find /proc/*/fd -lname 'anon_inode:inotify' 2>/dev/null | wc -l

# 3. If limits are set but Code-Server still shows the error:
docker restart Code-Server
# Running containers inherit limits at launch. Restart picks up the new values.

# 4. If limits are NOT set (inotify_tuning.sh hasn't run yet):
inotify_tuning.sh --log

---

docker_syslog_filter.sh

What It Creates

/etc/rsyslog.d/ignore-docker-veth.conf:
  if ($msg contains "veth" or $msg contains "docker0") then {
      stop
  }

This drops any syslog message containing "veth" or "docker0" before it reaches any output target, including the log file.

Verify

docker_syslog_filter.sh --status
# Shows filter file content and rsyslog process state

# Manual verify:
cat /etc/rsyslog.d/ignore-docker-veth.conf
pgrep -x rsyslogd && echo "rsyslog running" || echo "rsyslog NOT running"

# Test the filter is active (should produce no syslog output):
logger "test veth message"
grep "test veth" /var/log/syslog 2>/dev/null || echo "filtered correctly"

If Syslog Still Has Veth Noise

# 1. Verify filter file exists with correct content:
docker_syslog_filter.sh --status

# 2. If content differs — re-apply:
docker_syslog_filter.sh --log

# 3. Verify rsyslog is using the conf.d directory:
grep -r "IncludeConfig" /etc/rsyslog.conf
# Expected: IncludeConfig /etc/rsyslog.d/*.conf  (or similar)

---

clear_logs.sh

Thresholds

LOG_MIN_SIZE_MB=10       # skip system log if under this — keep recent history
LOG_DOCKER_MAX_MB=100    # clear Docker container log only if over this

LOG_FILES=(
    "/var/log/syslog"
    "/var/log/messages"
    "/var/log/dmesg"
)

Why Truncation Not Deletion

unRAID writes logs to tmpfs (/var/log). Truncation (: > file) keeps the file descriptor open and valid while emptying content — syslogd continues writing to the same fd without interruption. Deleting the file would orphan the file descriptor and syslog would stop writing until restarted.

Identifying Large Docker Logs

clear_logs.sh --status
# Shows top 10 Docker logs by size, current size vs threshold

# Find the biggest log manually:
du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5

---

rsync_stop.sh

Auto-Detection Logic

rsync_stop.sh detects whether an orchestrator script (daily/weekly/critical sync) is the parent of the running rsync process by scanning lock files in $LOCK_DIR.

Default behavior (orchestrator detected): Kills only the rsync subprocess. The orchestrator sees rsync died, moves to the next share or exits cleanly. The orchestrator is NOT killed — it can still clean up.

Default behavior (no orchestrator): Kills rsync directly (standalone rsync.sh run).

--full-stop: Kills the orchestrator first, then kills rsync. Nothing continues after this. Use when everything needs to stop immediately.

Container Recovery

After killing rsync, the script checks all containers in PROFILE_CRITICAL_CONTAINER_NAMES for any that were stopped by the interrupted rsync session and restarts them. Remote containers are left for docker_watchdog.sh to recover.

Skip container recovery with --rsync-only — used when called by array_stopping.sh (which handles container stop separately via docker_container_stop.sh).

Usage

rsync_stop.sh                        # smart stop (auto-detect orchestrator)
rsync_stop.sh --full-stop            # kill orchestrator + rsync
rsync_stop.sh --rsync-only           # kill rsync, skip container recovery
rsync_stop.sh --status               # show local and remote rsync state
rsync_stop.sh --dry-run              # preview without changes
rsync_stop.sh --full-stop --dry-run  # preview full stop

---

server_reboot.sh

Full Shutdown Sequence

1. Pre-flight checks (warn only — do not block):
   - rsync running?     → warn, suggest rsync_stop.sh first
   - mover running?     → warn, suggest mover_stop.sh first
   - Emby sessions?     → warn (active streams will be interrupted)

2. Wall message to all logged-in terminal users

3. unRAID dashboard notification

4. Wait REBOOT_SLEEP seconds (default: 30)

5. Graceful VM shutdown:
   virsh shutdown <each VM>  (ACPI signal — clean shutdown)
   Wait REBOOT_VM_WAIT seconds (default: 30) for VMs to respond

6. /etc/rc.d/rc.libvirt stop   (VM Manager)

7. Orchestrators/array_stopping.sh — safe ordered array stop:
   - conf_cache_save.sh           snapshot partner conf RAM cache
   - user_scripts_stop.sh         stop background User Scripts
   - fallback.sh --stop           graceful fallback teardown
   - rsync_stop.sh --rsync-only   kill active rsync transfers
   - mover_stop.sh                stop mover
   - docker_container_stop.sh     stop all containers gracefully

8. sync   (flush filesystem buffers to disk)

9. /sbin/reboot

Configuration

REBOOT_SLEEP=30      # seconds between warning and shutdown sequence
REBOOT_VM_WAIT=30    # seconds to wait for VMs to shut down gracefully

Recommended Pre-Reboot Sequence

For a clean reboot when services are active:

rsync_stop.sh          # stop any active rsync (smart mode)
server_reboot.sh --status  # check what's still running
server_reboot.sh --reason="planned maintenance"

Usage

server_reboot.sh                         # reboot with 30s warning
server_reboot.sh --dry-run               # walk through without rebooting
server_reboot.sh --status                # show what would be affected
server_reboot.sh --reason="disk work"    # include reason in notification

---

Full Configuration Reference

# master.conf

# ── Conf Sync ──────────────────────────────────────────────────────────────────
CONF_SYNC_ENABLED=true          # toggle: false disables conf_sync.sh entirely

# ── inotify Tuning ─────────────────────────────────────────────────────────────
INOTIFY_MAX_INSTANCES=1024
INOTIFY_MAX_WATCHES=1048576
INOTIFY_MAX_QUEUED_EVENTS=32768
INOTIFY_WARN_PCT=80             # warn (in reports) if instances exceed this % of limit

# ── Syslog Filter ──────────────────────────────────────────────────────────────
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"

# ── Log Cleaner ────────────────────────────────────────────────────────────────
LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
LOG_MIN_SIZE_MB=10
LOG_DOCKER_MAX_MB=100

# ── Server Reboot ──────────────────────────────────────────────────────────────
REBOOT_SLEEP=30
REBOOT_VM_WAIT=30

# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
STATE_DIR="${SCRIPTS_DIR}/State_Files"
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"

---

Troubleshooting

Partner Conf Not In RAM Cache After Boot

# Check what's in the RAM cache:
ls -la /tmp/.cache/vv/d/

# Check what's in the persistent backup:
ls -la "$PERSISTENT_CONF_CACHE/"    # set SCRIPTS_DIR first or use full path

# Re-run conf sync manually:
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log

# If partner is reachable but pull failed, check SSH key:
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"

Persistent Backup Is Stale or Empty

# conf_cache_watchdog.sh refreshes the backup while partner is offline.
# Check watchdog state:
cat "$STATE_DIR/conf_cache_watchdog_state.db"

# Force a manual backup from current RAM cache:
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log

rsync_stop Killed the Wrong Thing

If --full-stop killed an orchestrator you didn't intend to kill:

# Next time use default mode (no --full-stop) to kill only rsync subprocess.
# To verify what would be killed before running:
rsync_stop.sh --status       # shows running rsync and detected orchestrators
rsync_stop.sh --dry-run      # shows smart mode decision
rsync_stop.sh --full-stop --dry-run  # shows full-stop decision

inotify Exhaustion After Boot

# Verify limits are applied:
sysctl fs.inotify.max_user_watches    # expect 1048576
sysctl fs.inotify.max_user_instances  # expect 1024

# If not set — run manually:
inotify_tuning.sh --log

# Check current usage:
inotify_tuning.sh --status

Syslog Still Noisy After Array Start

# Check filter is in place:
docker_syslog_filter.sh --status

# Re-apply if needed:
docker_syslog_filter.sh --log

# Restart rsyslog to pick up the filter:
/etc/rc.d/rc.rsyslogd restart