Storage-mode awareness pass + doc update for System_Essentials through Partnership
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR / PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the ecosystem works in both internal and appdata storage modes. PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php, config.php): all state reads switched to STATE_DIR constant; remote state reads use the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via their varaverk.cfg before building the path. conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added _remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull. fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED, FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card. README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync, Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync, conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener, upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to $STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js, not PHP -S).
This commit is contained in:
Gmer4Lfe
@@ -4,23 +4,24 @@ Configuration reference, operational procedures, and troubleshooting for
|
||||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||||
adding or reordering scripts at array start.
|
||||
|
||||
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
|
||||
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
|
||||
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
|
||||
> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
|
||||
> `mover_stop.sh`, `user_scripts_stop.sh`) and their configuration live in
|
||||
> `Plugin/unraid/System_Essentials/`. Watchdog configuration (`stability_watchdog.sh`,
|
||||
> `resource_watchdog.sh`, `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives
|
||||
> in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [ARRAY_START_SCRIPTS Order](#array_start_scripts-order)
|
||||
- [webgui_watchdog.sh](#webgui_watchdogsh)
|
||||
- [ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order](#array_start_scripts--array_stop_scripts-order)
|
||||
- [conf_sync.sh](#conf_syncsh)
|
||||
- [conf_cache_save.sh](#conf_cache_savesh)
|
||||
- [conf_cache_restore.sh](#conf_cache_restoresh)
|
||||
- [inotify_tuning.sh](#inotify_tuningsh)
|
||||
- [php_fpm_max_children.sh](#php_fpm_max_childrensh)
|
||||
- [docker_syslog_filter.sh](#docker_syslog_filtersh)
|
||||
- [clear_logs.sh](#clear_logssh)
|
||||
- [mover_stop.sh](#mover_stopsh)
|
||||
- [rsync_stop.sh](#rsync_stopsh)
|
||||
- [user_scripts_stop.sh](#user_scripts_stopsh)
|
||||
- [server_reboot.sh](#server_rebootsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
@@ -32,109 +33,155 @@ adding or reordering scripts at array start.
|
||||
All scripts use a two-tier output model: `echo` lines are always visible; `log`
|
||||
lines only appear when `--log` is passed.
|
||||
|
||||
**Daemon scripts** (`webgui_watchdog.sh`): run on every cycle. Without `--log`, only
|
||||
state transitions, warnings, errors, and the clean-cycle conclusion line are visible.
|
||||
Per-check detail suppressed.
|
||||
|
||||
**One-shot scripts** (`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`,
|
||||
`mover_stop.sh`, `php_fpm_max_children.sh`, `rsync_stop.sh`, `server_reboot.sh`,
|
||||
`user_scripts_stop.sh`): without `--log`, section headers, per-step results, and the
|
||||
**One-shot scripts** (`conf_sync.sh`, `conf_cache_save.sh`, `conf_cache_restore.sh`,
|
||||
`clear_logs.sh`, `docker_syslog_filter.sh`, `inotify_tuning.sh`, `rsync_stop.sh`,
|
||||
`server_reboot.sh`): without `--log`, section headers, per-step results, and the
|
||||
final summary are visible. Per-item detail suppressed.
|
||||
|
||||
---
|
||||
|
||||
## ARRAY_START_SCRIPTS Order
|
||||
## ARRAY_START_SCRIPTS / ARRAY_STOP_SCRIPTS Order
|
||||
|
||||
> **The order of scripts in ARRAY_START_SCRIPTS matters for three of these
|
||||
> scripts.** Getting it wrong causes subtle failures that don't show up
|
||||
> immediately.
|
||||
> **Order matters.** Scripts that set up conditions other scripts depend on must run
|
||||
> first. Getting the order wrong causes subtle failures.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# master.conf (current order — from ARRAY_START_SCRIPTS)
|
||||
ARRAY_START_SCRIPTS=(
|
||||
"inotify_tuning.sh" # 1 — FIRST: kernel limits must be set before
|
||||
# any container starts. Containers inherit
|
||||
# inotify limits at launch, not dynamically.
|
||||
"docker_syslog_filter.sh" # 2 — SECOND: before any veth interfaces are
|
||||
# created. If a container starts first, its
|
||||
# veth creation is already in syslog.
|
||||
"php_fpm_max_children.sh" # 3 — before WebGUI is under load
|
||||
"ramdisk_setup.sh" # (from Transcodes/) before Emby starts
|
||||
...
|
||||
"Plugin/unraid/System_Essentials/unraid_api_key_renew.sh" # plugin API key — ephemeral registry
|
||||
"System_Essentials/conf_sync.sh" # 1st cross-server op — pull partner confs into RAM cache
|
||||
"System_Essentials/conf_cache_restore.sh" # fill gaps if partner was down at boot
|
||||
"Transcodes/ramdisk_setup.sh" # ramdisk before Emby starts
|
||||
"System_Essentials/docker_syslog_filter.sh" # before veth interfaces are created
|
||||
"Plugin/unraid/System_Essentials/php_fpm_max_children.sh" # WebGUI tuning
|
||||
"System_Essentials/inotify_tuning.sh" # before docker_network_connect and continuous scripts
|
||||
"Docker_Essentials/docker_network_connect.sh" # ensure networks + container connections
|
||||
"Tools/claude_startup.sh" # persist Claude data to appdata; re-symlink on boot
|
||||
"Media/start_webhook_listener.sh" # arr upgrade webhook — continuous
|
||||
"Fallback/fallback.sh" # mutual failover — continuous
|
||||
)
|
||||
|
||||
ARRAY_STOP_SCRIPTS=(
|
||||
"System_Essentials/conf_cache_save.sh" # FIRST: snapshot RAM cache while fresh
|
||||
"Plugin/unraid/System_Essentials/user_scripts_stop.sh"
|
||||
"Fallback/fallback.sh --stop"
|
||||
"System_Essentials/rsync_stop.sh --rsync-only"
|
||||
"Plugin/unraid/System_Essentials/mover_stop.sh"
|
||||
"Docker_Essentials/docker_container_stop.sh"
|
||||
)
|
||||
# Watchdogs are NOT in ARRAY_START_SCRIPTS — they run every 15 minutes via
|
||||
# Orchestrators/watchdog_orchestrator.sh (separate cron entry).
|
||||
```
|
||||
|
||||
Why inotify FIRST: If Code-Server starts before limits are raised, it inherits
|
||||
the old low limits. The limits are kernel-wide — a restart of Code-Server picks
|
||||
up the new values, but it's a manual step. Avoid by running inotify_tuning.sh first.
|
||||
Why conf_sync before conf_cache_restore: the sync runs first to get the freshest
|
||||
possible partner conf. The restore only fills in for confs that the sync couldn't
|
||||
fetch — it is a fallback, not the primary path.
|
||||
|
||||
Why docker_syslog_filter SECOND: The filter must be in place before any container
|
||||
starts creating veth interfaces. The first container start after array start
|
||||
generates veth messages — these will appear in syslog if the filter isn't active.
|
||||
Why docker_syslog_filter before inotify: the filter must be in place before any
|
||||
container creates veth interfaces. inotify must be set before docker_network_connect
|
||||
and the continuous scripts (start_webhook_listener, fallback), which are the first
|
||||
steps that actually touch containers.
|
||||
|
||||
Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
|
||||
`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
|
||||
else shuts down or changes state.
|
||||
|
||||
---
|
||||
|
||||
## webgui_watchdog.sh
|
||||
## conf_sync.sh
|
||||
|
||||
### Escalation Logic
|
||||
### Modes
|
||||
|
||||
```
|
||||
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
|
||||
|
||||
Not responding:
|
||||
1. platform_restart_service nginx
|
||||
wait WEBGUI_NGINX_WAIT (15s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
2. platform_restart_service php-fpm
|
||||
wait WEBGUI_PHP_WAIT (10s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
3. platform_restart_service emhttp
|
||||
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
|
||||
→ recovered: notify, exit 0
|
||||
|
||||
All three failed → notify warning, exit 1
|
||||
conf_sync.sh Full sync: pull from all partners + push to all partners
|
||||
conf_sync.sh --push-only Push own conf to all partners (fast — for conf-save hook)
|
||||
conf_sync.sh --pull-only Pull partner confs only (intermediate orchestrator)
|
||||
conf_sync.sh --dry-run Show what would happen, no changes
|
||||
conf_sync.sh --log Verbose output
|
||||
```
|
||||
|
||||
### Configuration
|
||||
### What It Syncs
|
||||
|
||||
- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
|
||||
via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
|
||||
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
|
||||
`/tmp/.cache/vv/d/${my_id}.conf` via SCP
|
||||
- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
|
||||
on full sync (so the cache has a complete snapshot of all confs)
|
||||
|
||||
Only partner confs are sourced from cache — `load_config.sh` always reads own conf
|
||||
from disk to avoid sourcing a stale cached copy.
|
||||
|
||||
### Remote SCRIPTS_DIR Resolution
|
||||
|
||||
The pull path reads the partner's `/boot/config/plugins/varaverk/varaverk.cfg` to
|
||||
find their actual `SCRIPTS_DIR` before building the SCP path. This handles the case
|
||||
where the partner is in appdata storage mode and their conf is at
|
||||
`/mnt/user/appdata/Varaverk/Configurations/` rather than the internal path.
|
||||
|
||||
### PARTNERSHIP_ENABLED Gate
|
||||
|
||||
conf_sync.sh calls `require_partnership` — if `PARTNERSHIP_ENABLED=false`, it exits
|
||||
silently with 0. The conf cache will be empty for partner confs while partnership
|
||||
is disabled.
|
||||
|
||||
### If Conf Pull Fails at Boot
|
||||
|
||||
If the partner is unreachable, `conf_sync.sh` logs a warning and exits 1.
|
||||
`conf_cache_restore.sh` then runs (next in ARRAY_START_SCRIPTS) and loads the
|
||||
persistent backup from `$PERSISTENT_CONF_CACHE` if available.
|
||||
|
||||
A notification fires if any partner fails — check partner reachability via Tailscale.
|
||||
|
||||
---
|
||||
|
||||
## conf_cache_save.sh
|
||||
|
||||
### What It Does
|
||||
|
||||
At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
|
||||
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
|
||||
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
|
||||
can't reach the partner.
|
||||
|
||||
`conf_cache_watchdog.sh` (in `Watchdogs/System/`) refreshes this backup every
|
||||
15 minutes while the partner is offline — keeping it current even during extended
|
||||
outages.
|
||||
|
||||
### PARTNERSHIP_ENABLED Gate
|
||||
|
||||
Exits silently when `PARTNERSHIP_ENABLED=false`. No backup is written.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
WEBGUI_URL="http://localhost" # URL to check
|
||||
WEBGUI_TIMEOUT=5 # curl timeout in seconds
|
||||
WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck
|
||||
WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck
|
||||
WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
|
||||
conf_cache_save.sh # save partner confs from RAM cache (called at array stop)
|
||||
conf_cache_save.sh --dry-run # show what would be saved
|
||||
conf_cache_save.sh --log # verbose output
|
||||
```
|
||||
|
||||
### WebGUI Frozen — Manual Recovery
|
||||
---
|
||||
|
||||
## conf_cache_restore.sh
|
||||
|
||||
### What It Does
|
||||
|
||||
At array start (after `conf_sync.sh`), checks which partner confs are missing from
|
||||
`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
|
||||
if a backup exists there.
|
||||
|
||||
Always removes the persistent backup when done — whether used or not. On a normal
|
||||
reboot where the partner was reachable, the sync already populated the cache and the
|
||||
backup is cleaned up without being used.
|
||||
|
||||
### PARTNERSHIP_ENABLED Gate
|
||||
|
||||
Exits silently when `PARTNERSHIP_ENABLED=false`.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Check which services are running:
|
||||
webgui_watchdog.sh --status
|
||||
|
||||
# Try manual restart sequence (mirrors what the script calls):
|
||||
# Source the ecosystem first to get platform functions:
|
||||
source /boot/config/plugins/varaverk/load_config.sh
|
||||
platform_restart_service nginx
|
||||
# wait 15s, then:
|
||||
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
|
||||
|
||||
# If nginx did not fix it, php-fpm:
|
||||
platform_restart_service php-fpm
|
||||
|
||||
# If still down, emhttp:
|
||||
platform_restart_service emhttp
|
||||
|
||||
# Raw equivalents (no source needed — paste directly into terminal):
|
||||
# /etc/rc.d/rc.nginx restart
|
||||
# /etc/rc.d/rc.php-fpm restart
|
||||
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
|
||||
|
||||
# If all three failed:
|
||||
server_reboot.sh --status # check for active sessions first
|
||||
conf_cache_restore.sh # restore missing confs from backup (called at array start)
|
||||
conf_cache_restore.sh --dry-run # show what would be restored
|
||||
conf_cache_restore.sh --log # verbose output
|
||||
```
|
||||
|
||||
---
|
||||
@@ -176,48 +223,6 @@ inotify_tuning.sh --log
|
||||
|
||||
---
|
||||
|
||||
## php_fpm_max_children.sh
|
||||
|
||||
### What It Sets
|
||||
|
||||
```bash
|
||||
PHP_MAX_CHILDREN=250 # target pm.max_children (default: 4-8 on unRAID)
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
```
|
||||
|
||||
250 workers × ~2MB per worker = ~500MB total. On 128GB this is trivially small.
|
||||
The default of 4–8 saturates immediately under load on a busy server.
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
php_fpm_max_children.sh --status
|
||||
# Shows current value vs target, PHP-FPM worker count
|
||||
|
||||
# Manual verify:
|
||||
grep "^pm.max_children" /etc/php83/php-fpm.d/www.conf
|
||||
# Expected: pm.max_children = 250
|
||||
```
|
||||
|
||||
### If WebGUI Is Slow Despite the Setting
|
||||
|
||||
```bash
|
||||
# Check PHP-FPM worker utilization (requires system_tuning_monitor.sh in Monitors/):
|
||||
# Look at the webgui_watchdog.sh escalation — step 2 (php-fpm restart) is specifically
|
||||
# for worker exhaustion. If webgui_watchdog.sh is regularly hitting step 2, the
|
||||
# pm.max_children value may still be too low, or there's a PHP worker leak.
|
||||
|
||||
# Check running worker count:
|
||||
pgrep -fc php-fpm
|
||||
# Compare to pm.max_children — if equal, workers are saturated
|
||||
|
||||
# Increase if needed:
|
||||
# master.conf: PHP_MAX_CHILDREN=350
|
||||
# Then: php_fpm_max_children.sh --log (will update and restart php-fpm)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## docker_syslog_filter.sh
|
||||
|
||||
### What It Creates
|
||||
@@ -230,8 +235,7 @@ pgrep -fc php-fpm
|
||||
```
|
||||
|
||||
This drops any syslog message containing "veth" or "docker0" before it reaches
|
||||
any output target, including the log file. The drop rule is applied at rsyslog
|
||||
level — not at the log viewer level.
|
||||
any output target, including the log file.
|
||||
|
||||
### Verify
|
||||
|
||||
@@ -298,39 +302,6 @@ du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
|
||||
|
||||
---
|
||||
|
||||
## mover_stop.sh
|
||||
|
||||
### Stop Sequence
|
||||
|
||||
```
|
||||
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
|
||||
2. Wall message to all logged-in terminal users
|
||||
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
|
||||
4. SIGTERM — mover finishes its current file operation, then stops
|
||||
5. Wait 5 seconds → verify stopped
|
||||
6. SIGKILL if still running — forced stop, partial files possible
|
||||
7. Final verify — error if still running after SIGKILL
|
||||
```
|
||||
|
||||
SIGTERM first because the mover can finish the file it is currently moving,
|
||||
leaving no partial copies split across cache and array. SIGKILL is a last resort.
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
MOVER_STOP_TIMEOUT=30 # seconds between wall warning and SIGTERM
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
mover_stop.sh # check and stop if running
|
||||
mover_stop.sh --status # show current mover state and PID
|
||||
mover_stop.sh --dry-run # show what would happen without stopping
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## rsync_stop.sh
|
||||
|
||||
### Auto-Detection Logic
|
||||
@@ -355,8 +326,8 @@ After killing rsync, the script checks all containers in `PROFILE_CRITICAL_CONTA
|
||||
for any that were stopped by the interrupted rsync session and restarts them.
|
||||
Remote containers are left for docker_watchdog.sh to recover.
|
||||
|
||||
Skip container recovery with `--rsync-only` — used when called by other scripts
|
||||
that handle recovery themselves.
|
||||
Skip container recovery with `--rsync-only` — used when called by array_stopping.sh
|
||||
(which handles container stop separately via docker_container_stop.sh).
|
||||
|
||||
### Usage
|
||||
|
||||
@@ -371,34 +342,6 @@ rsync_stop.sh --full-stop --dry-run # preview full stop
|
||||
|
||||
---
|
||||
|
||||
## user_scripts_stop.sh
|
||||
|
||||
### Process Identification
|
||||
|
||||
Scans `/proc/*/cmdline` for any process whose command line contains
|
||||
`/tmp/user.scripts`. The unRAID User Scripts plugin stages all scripts in
|
||||
`/tmp/user.scripts/` before execution — this signature is reliable regardless of
|
||||
what the script is named or how it was launched.
|
||||
|
||||
Script names are extracted from the path for display: you see which scripts are
|
||||
being stopped, not just PIDs.
|
||||
|
||||
### Self-Exclusion
|
||||
|
||||
If this script is run via the User Scripts plugin, it would find its own PID in
|
||||
the scan. It excludes both `$$` (its own PID) and `$PPID` (its parent process)
|
||||
from the kill list.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all User Script processes
|
||||
user_scripts_stop.sh --status # show running scripts with names and elapsed time
|
||||
user_scripts_stop.sh --dry-run # show what would be stopped
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## server_reboot.sh
|
||||
|
||||
### Full Shutdown Sequence
|
||||
@@ -422,11 +365,12 @@ user_scripts_stop.sh --dry-run # show what would be stopped
|
||||
6. /etc/rc.d/rc.libvirt stop (VM Manager)
|
||||
|
||||
7. Orchestrators/array_stopping.sh — safe ordered array stop:
|
||||
- user_scripts_stop.sh stop background User Scripts
|
||||
- fallback.sh --stop graceful fallback teardown
|
||||
- rsync_stop.sh --rsync-only kill active rsync transfers
|
||||
- mover_stop.sh stop mover
|
||||
- docker_container_stop.sh stop all containers gracefully
|
||||
- conf_cache_save.sh snapshot partner conf RAM cache
|
||||
- user_scripts_stop.sh stop background User Scripts
|
||||
- fallback.sh --stop graceful fallback teardown
|
||||
- rsync_stop.sh --rsync-only kill active rsync transfers
|
||||
- mover_stop.sh stop mover
|
||||
- docker_container_stop.sh stop all containers gracefully
|
||||
|
||||
8. sync (flush filesystem buffers to disk)
|
||||
|
||||
@@ -446,7 +390,6 @@ For a clean reboot when services are active:
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # stop any active rsync (smart mode)
|
||||
mover_stop.sh # stop mover gracefully
|
||||
server_reboot.sh --status # check what's still running
|
||||
server_reboot.sh --reason="planned maintenance"
|
||||
```
|
||||
@@ -464,27 +407,17 @@ server_reboot.sh --reason="disk work" # include reason in notification
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
|
||||
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
|
||||
# ── WebGUI Watchdog ────────────────────────────────────────────────────────────
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
WEBGUI_NGINX_WAIT=15
|
||||
WEBGUI_PHP_WAIT=10
|
||||
WEBGUI_EMHTTP_WAIT=30
|
||||
# ── Conf Sync ──────────────────────────────────────────────────────────────────
|
||||
CONF_SYNC_ENABLED=true # toggle: false disables conf_sync.sh entirely
|
||||
|
||||
# ── inotify Tuning ─────────────────────────────────────────────────────────────
|
||||
INOTIFY_MAX_INSTANCES=1024
|
||||
INOTIFY_MAX_WATCHES=1048576
|
||||
INOTIFY_MAX_QUEUED_EVENTS=32768
|
||||
|
||||
# ── PHP-FPM ────────────────────────────────────────────────────────────────────
|
||||
PHP_MAX_CHILDREN=250
|
||||
PHP_CONF="/etc/php83/php-fpm.d/www.conf"
|
||||
INOTIFY_WARN_PCT=80 # warn (in reports) if instances exceed this % of limit
|
||||
|
||||
# ── Syslog Filter ──────────────────────────────────────────────────────────────
|
||||
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
||||
@@ -494,20 +427,48 @@ LOG_FILES=("/var/log/syslog" "/var/log/messages" "/var/log/dmesg")
|
||||
LOG_MIN_SIZE_MB=10
|
||||
LOG_DOCKER_MAX_MB=100
|
||||
|
||||
# ── Mover Stop ─────────────────────────────────────────────────────────────────
|
||||
MOVER_STOP_TIMEOUT=30
|
||||
|
||||
# ── Server Reboot ──────────────────────────────────────────────────────────────
|
||||
REBOOT_SLEEP=30
|
||||
REBOOT_VM_WAIT=30
|
||||
|
||||
# ── State Paths (auto-adapt to storage mode) ───────────────────────────────────
|
||||
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
|
||||
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
|
||||
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
|
||||
STATE_DIR="${SCRIPTS_DIR}/State_Files"
|
||||
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
|
||||
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
|
||||
### Partner Conf Not In RAM Cache After Boot
|
||||
|
||||
```bash
|
||||
# Check what's in the RAM cache:
|
||||
ls -la /tmp/.cache/vv/d/
|
||||
|
||||
# Check what's in the persistent backup:
|
||||
ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path
|
||||
|
||||
# Re-run conf sync manually:
|
||||
/boot/config/plugins/varaverk/System_Essentials/conf_sync.sh --log
|
||||
|
||||
# If partner is reachable but pull failed, check SSH key:
|
||||
ssh -i /path/to/ssh_key root@<partner-tailscale-ip> "echo ok"
|
||||
```
|
||||
|
||||
### Persistent Backup Is Stale or Empty
|
||||
|
||||
```bash
|
||||
# conf_cache_watchdog.sh refreshes the backup while partner is offline.
|
||||
# Check watchdog state:
|
||||
cat "$STATE_DIR/conf_cache_watchdog_state.db"
|
||||
|
||||
# Force a manual backup from current RAM cache:
|
||||
/boot/config/plugins/varaverk/System_Essentials/conf_cache_save.sh --log
|
||||
```
|
||||
|
||||
### rsync_stop Killed the Wrong Thing
|
||||
|
||||
@@ -521,36 +482,29 @@ rsync_stop.sh --dry-run # shows smart mode decision
|
||||
rsync_stop.sh --full-stop --dry-run # shows full-stop decision
|
||||
```
|
||||
|
||||
### WebGUI Recovery After All Three Steps Failed
|
||||
### inotify Exhaustion After Boot
|
||||
|
||||
```bash
|
||||
# Check if processes are running:
|
||||
pgrep -x nginx && echo "nginx: yes" || echo "nginx: no"
|
||||
pgrep emhttpd && echo "emhttp: yes" || echo "emhttp: no"
|
||||
pgrep -f php-fpm && echo "php-fpm: yes" || echo "php-fpm: no"
|
||||
# Verify limits are applied:
|
||||
sysctl fs.inotify.max_user_watches # expect 1048576
|
||||
sysctl fs.inotify.max_user_instances # expect 1024
|
||||
|
||||
# Check recent nginx errors:
|
||||
cat /var/log/nginx/error.log | tail -20
|
||||
# If not set — run manually:
|
||||
inotify_tuning.sh --log
|
||||
|
||||
# Check emhttp log:
|
||||
tail -20 /var/log/syslog | grep emhttp
|
||||
|
||||
# Last resort — reboot:
|
||||
server_reboot.sh --reason="WebGUI unrecoverable"
|
||||
# Check current usage:
|
||||
inotify_tuning.sh --status
|
||||
```
|
||||
|
||||
### PHP-FPM Config Not Found After unRAID Update
|
||||
|
||||
unRAID updates occasionally change the PHP version. If `php_fpm_max_children.sh`
|
||||
errors with "config file not found":
|
||||
### Syslog Still Noisy After Array Start
|
||||
|
||||
```bash
|
||||
# Find the new config path:
|
||||
find /etc -name "www.conf" 2>/dev/null
|
||||
# Check filter is in place:
|
||||
docker_syslog_filter.sh --status
|
||||
|
||||
# Update PHP_CONF in master.conf:
|
||||
PHP_CONF="/etc/php84/php-fpm.d/www.conf" # example for php84
|
||||
# Re-apply if needed:
|
||||
docker_syslog_filter.sh --log
|
||||
|
||||
# Verify with:
|
||||
php_fpm_max_children.sh --status
|
||||
# Restart rsyslog to pick up the filter:
|
||||
/etc/rc.d/rc.rsyslogd restart
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user