# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 🖥️ UNRAID ESSENTIALS # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ **System-level scripts that act on the unRAID server itself — not containers, not media, not monitoring.** Keeping the server stable under load, recovering a frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful shutdowns and reboots with proper warning sequences. ``` unRAID_Essentials/ ← acts on the server itself (this folder) Docker_Essentials/ ← acts on containers Media/ ← acts on the library Monitors/ ← observes and reports ``` > **The escalation chain matters here.** Docker_Essentials handles container-level > problems. unRAID_Essentials handles server-level problems. The watchdogs are > designed to work together — docker_watchdog.sh heals containers first, > system_watchdog.sh reboots only when healing has failed. --- ## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ --- ### 🔴 Server Getting Into Unstable States With No Recovery Path A container has a severe memory leak. RAM drops to 2GB. The system starts swapping. Everything slows down. Docker watchdog tries to restart the container — but Docker itself is barely responding. The restart hangs. The watchdog is stuck. Nothing is getting better. The server needs a reboot, but nothing in the ecosystem is authorised to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The WebGUI shows nothing useful. The server is functionally dead but still technically running. Again — needs a reboot, nothing calls one. The fix: `system_watchdog.sh` — the last line of defense. Three-tier response: immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, and a strike system for sustained threshold breaches. When everything else has failed, system_watchdog reboots cleanly before a hard crash happens. --- ### 🔴 WebGUI Freezing and Nobody Noticing The WebGUI becomes unresponsive. Nginx gets into a bad state. Or PHP-FPM workers are saturated and new requests are queueing indefinitely. Or emhttp itself has frozen. From a user perspective: dashboard doesn't load, settings don't save, containers can't be started or stopped via the UI. Nothing in the container stack alerts on a frozen WebGUI — it's not a container problem, it's a web server problem. The only way to know is if someone tries to use the UI and notices. By which point it may have been broken for hours. The fix: `webgui_restart.sh` — checks every 10 minutes, escalates through nginx → php-fpm → emhttp. Lightest fix first. Notifies on any restart so you know it happened. Silent when healthy — 144 runs per day with no output is the correct behaviour. --- ### 🔴 50+ Containers Starting and Filling Syslog With Veth Noise Array starts. 50+ containers come up simultaneously. Docker creates a virtual network interface for each one. Each creation generates multiple syslog entries. In the first few minutes after array start the syslog is buried under hundreds of lines of: ``` kernel: veth2a3b4c5: renamed from eth0 kernel: docker0: port 1(veth2a3b4c5) entered blocking state kernel: docker0: port 1(veth2a3b4c5) entered forwarding state ``` Real events — a failed mount, a permission error, a service that didn't start — are invisible in this noise. And on a busy server that restarts containers regularly, this noise continues throughout the day. The fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container starts. Applied at array start, idempotent, silent when already correct. --- ### 🔴 WebGUI Queueing Requests Under Load Without Explanation The WebGUI feels slow. Clicking a button takes 5 seconds. Saving settings seems to hang. Nothing in the logs explains it. Container starts from the UI timeout. The server itself is not under load — CPU is fine, RAM is fine. The cause: PHP-FPM's `pm.max_children` defaults to 4-8 workers. On a server with multiple users, active plugins, automated tools polling the API, and 50+ containers all potentially hitting the WebGUI simultaneously, those 4-8 workers saturate immediately. New requests queue behind active ones. Everything feels slow. The fix: `php_fpm_max_children.sh` — sets `pm.max_children=250` at array start. 250 workers × ~40MB = ~10GB worst case. On 128GB this is trivially small. The WebGUI becomes responsive immediately. Idempotent — silent when already correct. --- ### 🔴 inotify Exhaustion Producing Unexplained Failures Already documented in README-Monitors.md (system_tuning_monitor.sh section). Short version: when inotify limits are exhausted, containers silently stop receiving file system events. Downloads complete but arrs don't detect them. The kernel hits the limit and new watches fail silently. VSCode shows "unable to watch for file changes" and misses edits. The fix: `inotify_tuning.sh` — raises all three inotify limits at array start. 1M watches (raised from 512K — Code-Server with node_modules needs this), 1024 instances, 32768 queued events. The startup race note: if Code-Server starts before this runs, it inherits old limits. Restart Code-Server if the VSCode error appears after limits are applied. --- ### 🔴 Mover Getting Killed Mid-Transfer Leaving Files Inconsistent The mover is running — moving a large batch of files from cache to array. Someone clicks reboot from the UI. Or a script kills the mover process directly. The mover stops mid-file. The file exists partially on both cache and array simultaneously. unRAID's deduplication layer is confused. The file is inaccessible. The fix: `mover_stop.sh` — warns logged-in users via wall message, waits the configured timeout, then sends SIGTERM (graceful) and verifies. The mover gets to finish its current file operation before stopping. SIGKILL is a last resort with a warning that partial files may exist. --- ## ━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Four distinct server-level roles: ``` 🛡️ Last-resort stability system_watchdog.sh — reboots before crash 🌐 WebGUI availability webgui_restart.sh — recovers frozen UI ⚙️ Kernel tuning inotify_tuning.sh — file watch limits php_fpm_max_children.sh — PHP worker count 🔇 Log hygiene docker_syslog_filter.sh — suppress veth noise clear_logs.sh — weekly log trimming 🔄 Graceful operations mover_stop.sh — clean mover stop server_reboot.sh — clean reboot with warning user_scripts_stop.sh — stop running scripts ``` --- ## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ | Script | Purpose | When | |--------|---------|------| | `system_watchdog.sh` | Three-tier last-resort stability watchdog | Continuous background loop | | `webgui_restart.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | Every 10 minutes | | `inotify_tuning.sh` | Raise inotify kernel limits | At array start | | `php_fpm_max_children.sh` | Set PHP-FPM worker count | At array start | | `docker_syslog_filter.sh` | Suppress Docker veth log noise | At array start | | `clear_logs.sh` | Size-threshold weekly log cleanup | Weekly via maintenance window | | `mover_stop.sh` | Clean mover stop with SIGTERM → SIGKILL | Manual | | `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual or called by system_watchdog | | `user_scripts_stop.sh` | Stop all running User Script processes | Manual or called by server_reboot | --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🛡️ system_watchdog.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ The last line of defense. Reboots the system cleanly before it crashes uncleanly. Three-tier response system — critical failures bypass everything and reboot immediately, OOM-confirmed crises bypass the strike system, sustained threshold breaches use strikes. Runs continuously as a background process started by `array_start.sh`. > Full architecture documentation in `README-Docker_Essentials.md` under > "Relationship to System Watchdog". This section covers the system_watchdog itself. ```bash # Started by: array_start.sh (continuous background process) # Interval: SYSTEM_WATCHDOG_INTERVAL=300 (5 minutes) ``` --- ### ── Three-Tier Response System ────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # TIER 1 — CRITICAL (bypass ALL strikes, reboot immediately) # These failures are acute — the system is not recoverable by waiting. # Single detection = immediate reboot. No confirmation window. # # Docker daemon unresponsive: # Attempt /etc/rc.d/rc.docker restart first. # Wait 15 seconds. Verify daemon responding. # If still hung → CRITICAL reboot. # A hung daemon cannot be healed — every subsequent docker command hangs. # # rootfs at ROOTFS_CRITICAL_PCT (99%+): # Writes are failing. SSH may stop. Logs can't be written. # Nothing can be fixed from this state without a reboot. # # Kernel BUG/Oops in dmesg: # Kernel running with corrupted state. # Delta-based: new oops since last cycle → reboot. # # File descriptor exhaustion (FD_CRITICAL_PCT=95%): # New connections failing. Docker can't spawn processes. SSH drops. # # /boot read-only: # State files and config writes silently failing. # Write test on /boot every cycle. # TIER 2 — URGENT (bypass strikes when OOM confirms crisis) # RAM < MEM_GB (4GB) AND OOM kills >= OOM_LIMIT (3) in this cycle # Without OOM confirmation → standard strike system applies. # Rationale: 1-2 OOM kills = docker_watchdog.sh handles it. # 3+ kills while RAM critical = system dying faster than watchdogs heal. # OOM victims from dmesg included in reboot message (diagnostic context). # TIER 3 — STANDARD (strike system — N consecutive failures → reboot) # Everything else: RAM tiers, load, CPU temp, zombies, /var/log, # /tmp, containers, NIC, mdstat, sshd # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── RAM Tiers ──────────────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # Three-level graduated response — not a single threshold. # SYS_WATCHDOG_MEM_WARN_GB=10 # warn + notify once — informational only SYS_WATCHDOG_MEM_SHUTDOWN_GB=6 # stop non-essential containers, wait for recovery SYS_WATCHDOG_MEM_GB=4 # strike system → reboot (or bypass if OOM confirms) SYS_WATCHDOG_MEM_RECOVER_GB=30 # RAM must reach this before restarting containers # # Containers excluded from RAM emergency shutdown — these stay running: SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED=( "NginxProxyManager" # external access — stop this and users lose everything "Authelia" # auth — without this nothing is accessible "Mariadb" # Authelia dependency "Redis" # Authelia dependency "Emby" # media server — Live TV buffering "Dispatcharr" # Live TV scheduler — loses state if stopped ) # # Coordination with docker_watchdog.sh: # system_watchdog writes mem_shutdown_active=true to SYS_WATCHDOG_STATE_FILE. # docker_watchdog reads this flag and defers ALL container restart logic. # Without this: both watchdogs fight — system_watchdog stops containers, # docker_watchdog restarts them, RAM never recovers. # With this: docker_watchdog stands down until mem_shutdown_active clears. # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Per-Host Check Toggles ─────────────────────────────────────────────────── ```bash # master_host1.conf (all checks in master_host*.conf — not master.conf) # ───────────────────────────────────────────────────────────────────────────── # Each check is independently toggleable per server. # HOST1 and HOST2 may have different hardware and different workloads. # detect_hosts() aliases HOST*_SYS_WATCHDOG_CHECK_* → SYS_WATCHDOG_CHECK_* # # Tier 1 — Critical (bypass strikes): HOST1_SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true HOST1_SYS_WATCHDOG_CHECK_ROOTFS=true HOST1_SYS_WATCHDOG_CHECK_KERNEL_OOPS=true HOST1_SYS_WATCHDOG_CHECK_FD=true HOST1_SYS_WATCHDOG_CHECK_BOOT=true # Tier 2 — Urgent (OOM bypass): HOST1_SYS_WATCHDOG_CHECK_OOM=true HOST1_SYS_WATCHDOG_CHECK_RAM=true # Tier 3 — Standard (strike system): HOST1_SYS_WATCHDOG_CHECK_LOG=true HOST1_SYS_WATCHDOG_CHECK_ARC=true # HOST1 runs ZFS — enable HOST1_SYS_WATCHDOG_CHECK_CPU_TEMP=true HOST1_SYS_WATCHDOG_CHECK_LOAD=false # disabled — transcoding causes normal spikes HOST1_SYS_WATCHDOG_CHECK_ZOMBIES=true HOST1_SYS_WATCHDOG_CHECK_CONTAINERS=true # monitors docker_watchdog persistent skip list HOST1_SYS_WATCHDOG_CHECK_TMP=true HOST1_SYS_WATCHDOG_CHECK_MDSTAT=true HOST1_SYS_WATCHDOG_CHECK_NETWORK=true HOST1_SYS_WATCHDOG_CHECK_SSHD=true HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false # disabled — may false positive during encoding HOST1_SYS_WATCHDOG_NIC="eth0" # verify: ip link show | grep "^[0-9]" ``` --- ### ── Abort Conditions ───────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # Conditions that prevent a reboot even when a threshold is hit. # CRITICAL tier bypasses these — truly critical conditions reboot regardless. # SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # rebooting with bad pool risks data loss SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity is better than crashing SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting mover is better than crashing # # Philosophy: a graceful reboot before a crash is always better than a hard crash. # The abort conditions protect against the cases where a reboot itself causes harm # (data loss from bad ZFS pool). Parity and mover can be restarted after reboot. ``` --- ### ── Reboot Loop Protection ─────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # If the server keeps rebooting, something is wrong that rebooting isn't fixing. # After REBOOT_LIMIT reboots in REBOOT_WINDOW_HRS → shutdown instead. # Shutdown prevents: hardware damage, filesystem corruption from repeated reboots, # infinite loop that never lets you investigate. # State file: /boot/config/system_watchdog_reboots.db — survives reboots. # SYS_WATCHDOG_REBOOT_LIMIT=3 # reboots before shutdown instead SYS_WATCHDOG_REBOOT_WINDOW_HRS=12 # rolling window in hours ``` --- ### ── State File Heartbeat ───────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # system_watchdog writes watchdog_cycle=N to SYS_WATCHDOG_STATE_FILE every cycle. # This keeps the file's modification time current. # # docker_watchdog.sh uses the state file mtime as a stale guard — if the file # is more than 2 hours old while mem_shutdown_active=true is set, system_watchdog # may have stopped running. docker_watchdog resumes normal operation rather than # being silenced indefinitely by a stale flag. # # Without this heartbeat: if all standard checks pass and no state writes happen # (e.g. CHECK_KERNEL_OOPS=false AND CHECK_MDSTAT=false), the file mtime could go # stale even with the watchdog running. ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash system_watchdog.sh # normal (continuous — started by array_start.sh) system_watchdog.sh --dry-run # trigger detection without rebooting system_watchdog.sh --status # show all tiers, thresholds, active strikes, RAM state system_watchdog.sh --log # verbose per-cycle output ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🌐 webgui_restart.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Monitors the unRAID WebGUI availability and recovers it automatically when unresponsive. Three-step escalating strategy — lightest fix first, heaviest last. Silent when healthy. ```bash # Scheduled: */10 * * * * (every 10 minutes) ``` --- ### ── Three-Step Escalation ──────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Step 1 — nginx restart # Lightest fix — handles most WebGUI failures. # nginx is the web server layer. Crash, worker stuck, connection timeout. # Wait WEBGUI_NGINX_WAIT seconds → curl recheck. # # Step 2 — php-fpm restart # Added because WebGUI can appear frozen due to PHP worker exhaustion. # pm.max_children workers all occupied → new requests queue → dashboard hangs. # php-fpm restart far less disruptive than emhttp. # Wait WEBGUI_PHP_WAIT seconds → curl recheck. # Note: system_tuning_monitor.sh tracks worker saturation over time. # # Step 3 — emhttp restart # Heaviest fix. emhttp is the core unRAID management daemon. # Array, Docker, shares continue running — only WebGUI management restarts. # Takes longer to recover — WEBGUI_EMHTTP_WAIT gives it time. # Wait WEBGUI_EMHTTP_WAIT seconds → curl recheck. # # If all three fail → notify warning — manual intervention needed. # Guidance: pgrep -x nginx emhttp | check journalctl | consider server_reboot.sh # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Configuration ──────────────────────────────────────────────────────────── ```bash # master.conf WEBGUI_URL="http://localhost" # adjust if non-standard port WEBGUI_TIMEOUT=5 # seconds before curl times out WEBGUI_NGINX_WAIT=15 # seconds after nginx restart before recheck WEBGUI_PHP_WAIT=10 # seconds after php-fpm restart before recheck WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart — takes longer ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash webgui_restart.sh # check once — silent if healthy, escalates if not webgui_restart.sh --dry-run # walk through escalation without restarting anything webgui_restart.sh --status # show WebGUI state + nginx/php-fpm/emhttp process state webgui_restart.sh --log # verbose — show each escalation step ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 📡 inotify_tuning.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Raises Linux inotify kernel limits at array start. Idempotent — completely silent when values are already correct. ```bash # Scheduled: At Startup of Array (via array_start.sh — FIRST in ARRAY_START_SCRIPTS) # Must run before containers start — containers inherit limits at startup ``` --- ### ── Three Limits ───────────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── INOTIFY_MAX_INSTANCES=1024 # default: 128 — max inotify instances per user # 1024 handles ~20-30 containers watching files INOTIFY_MAX_WATCHES=1048576 # default: 8192 — SHARED budget across ALL users/containers # Was 512K — raised to 1M (1048576) # VSCode/Code-Server alone needs ~50K-200K for large workspaces # with node_modules. All arr containers + Emby + VSCode share # this budget. 1M safe on 128GB RAM (~128MB kernel memory) # If VSCode shows "unable to watch for file changes" → too low INOTIFY_MAX_QUEUED_EVENTS=32768 # default: 16384 — events buffered before dropping ``` --- ### ── Startup Race Condition ─────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # inotify limits are kernel-wide — they take effect immediately on sysctl write. # But containers that have ALREADY started inherit the OLD limits at startup. # Those containers keep their inherited (low) limits until restarted. # # This is why inotify_tuning.sh must be FIRST in ARRAY_START_SCRIPTS — before # any container starts. If Code-Server starts before limits are raised: # → Code-Server inherits old 8192 watch limit # → VSCode shows "unable to watch for file changes" # → Fix: docker restart Code-Server (picks up current kernel limits on start) # # The script warns if it changed any values: # "If Code-Server is running: docker restart Code-Server" # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash inotify_tuning.sh # normal run (idempotent — silent when correct) inotify_tuning.sh --dry-run # show what would change without changing inotify_tuning.sh --status # current values vs targets + top inotify consumers inotify_tuning.sh --log # verbose — show each sysctl write ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ⚙️ php_fpm_max_children.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Raises PHP-FPM `pm.max_children` at array start to prevent WebGUI queueing under load. Idempotent — completely silent when already correct. No PHP-FPM restart unless value actually changed. ```bash # Scheduled: At Startup of Array (via array_start.sh) ``` --- ### ── Why 250 Workers ────────────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── PHP_MAX_CHILDREN=250 # default: 4-8 — far too low for a busy server # Each worker: ~2MB resident memory at idle # 250 workers × 2MB = ~500MB — trivial on 128GB # Worst case (all active): ~250 × 40MB = ~10GB # In practice: rarely all active simultaneously # On 64GB (HOST2): still appropriate — 250 × 40MB # = 10GB worst case = 15% of RAM, acceptable # # Without this fix: # 5 users hit the WebGUI simultaneously → 8 workers exhausted → 5 more queue # Each queued request waits for a worker to free → 5-10 second response times # Looks like a slow server — it's just a queue # PHP_CONF="/etc/php83/php-fpm.d/www.conf" # path may change with PHP version ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash php_fpm_max_children.sh # normal run (idempotent) php_fpm_max_children.sh --dry-run # show what would change php_fpm_max_children.sh --status # show current value vs target + worker count php_fpm_max_children.sh --log # verbose — show config write + restart ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🔇 docker_syslog_filter.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Creates an rsyslog drop rule for Docker veth/docker0 interface noise. Idempotent — silent when filter already correct, only writes + restarts rsyslog when something changed. ```bash # Scheduled: At Startup of Array (via array_start.sh — before containers start) ``` --- ### ── What Gets Suppressed ───────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # These kernel messages are generated on every container start and stop: # # kernel: veth2a3b4c5: renamed from eth0 # kernel: docker0: port 1(veth2a3b4c5) entered blocking state # kernel: docker0: port 1(veth2a3b4c5) entered forwarding state # kernel: docker0: port 1(veth2a3b4c5) entered disabled state # # 50+ containers at array start = 200-400 lines of this in the first minute. # Containers restart throughout the day = continuous noise. # Real events buried and invisible in syslog. # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Idempotent Design ──────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # Runs at every array start but only changes something when needed: # Filter file exists and content is correct → exit 0 silently # Filter file missing or content changed → write + restart rsyslog # # Expected content compared exactly — single source of truth: EXPECTED_FILTER='if ($msg contains "veth" or $msg contains "docker0") then { stop }' # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash docker_syslog_filter.sh # normal run (idempotent — silent when correct) docker_syslog_filter.sh --dry-run # show what would be written without writing docker_syslog_filter.sh --status # show current filter file + rsyslog state ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🗑️ clear_logs.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Clears system logs and Docker container logs using size thresholds. Only clears logs large enough to be worth clearing — preserves recent diagnostic context on small logs. ```bash # Called by: weekly_sync_maintenance.sh via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am) ``` --- ### ── Size Threshold Approach ────────────────────────────────────────────────── ```bash # master.conf # ───────────────────────────────────────────────────────────────────────────── # Why thresholds instead of clearing everything: # A 2MB syslog contains useful recent history — not worth clearing. # A 500MB Docker log is consuming rootfs and contains mostly noise — clear it. # Blind truncation destroys diagnostic context for no benefit. # LOG_MIN_SIZE_MB=10 # skip system log if under this size — keep history LOG_DOCKER_MAX_MB=100 # clear Docker container log only if over this size # Active containers (Emby, SABnzbd) grow fastest # 100MB × 30 containers = 3GB before any clearing kicks in LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg) # # Truncation not logrotate: # unRAID writes to tmpfs (/var/log) — logrotate's compress + archive approach # would consume more tmpfs space, not less. # : > file keeps the file descriptor valid while emptying content. # Safe for running services (syslogd continues writing to the same fd). # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash clear_logs.sh # normal run — silent if all logs under threshold clear_logs.sh --dry-run # show what would be cleared and sizes clear_logs.sh --status # show current log sizes vs thresholds clear_logs.sh --log # verbose — show each file evaluated ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## ⏹️ mover_stop.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Stops the unRAID mover cleanly — wall warning, configurable timeout, SIGTERM → verify → SIGKILL sequence. Safe to run when mover is not running — exits cleanly with a log. --- ### ── Stop Sequence ──────────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # 1. Check if mover is running — exit cleanly if not # 2. Wall message: "HOST1 (unRAID-Gmer4Lfe) — mover stopping in 30s" # MY_ID included — on shared terminal it's clear which server # 3. Wait MOVER_STOP_TIMEOUT seconds # 4. SIGTERM — allows mover to finish its current file before stopping # No partial files — the mover completes what it's working on # 5. Wait 5 seconds — verify if stopped # 6. If still running → SIGKILL (forced) # Warning: partial files possible — same as a hard crash # 7. Final verify — error if still running # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash mover_stop.sh # stop mover with configured timeout mover_stop.sh --dry-run # show what would happen mover_stop.sh --status # show mover state (running, PID, start time) ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🔁 server_reboot.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Graceful reboot with pre-flight warnings, wall message, unRAID notification, VM graceful shutdown, then Docker and services stop, sync, reboot. > Full documentation in `README-Tools.md` — `server_reboot.sh` section. This is a > quick reference. --- ### ── Shutdown Sequence ──────────────────────────────────────────────────────── ```bash # ───────────────────────────────────────────────────────────────────────────── # 1. Pre-flight warnings (warn not block): # rsync running, mover running, active Emby sessions # Warnings show in summary — you chose to reboot, these are for context # 2. Wall message + unRAID notification — MY_ID included # 3. Wait REBOOT_SLEEP seconds (default 30) # 4. virsh shutdown each VM → wait REBOOT_VM_WAIT seconds for graceful exit # 5. Stop libvirt (VM Manager) # 6. Stop Docker service # 7. sync — flush filesystem buffers # 8. /sbin/reboot # ───────────────────────────────────────────────────────────────────────────── ``` --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash server_reboot.sh # reboot with 30s warning server_reboot.sh --dry-run # full sequence walkthrough without rebooting server_reboot.sh --status # show running processes that would be affected server_reboot.sh --reason="maintenance" # include reason in wall + notification server_reboot.sh --log # verbose per-step output ``` --- ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ## 🛑 user_scripts_stop.sh ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Stops all running User Script processes. Identifies by `/tmp/user.scripts` path signature. Shows script names not PIDs. SIGTERM → verify → SIGKILL with self-exclusion. > Full documentation in `README-Tools.md` — `user_scripts_stop.sh` section. --- ### ── Usage ─────────────────────────────────────────────────────────────────── ```bash user_scripts_stop.sh # stop all — SIGTERM → SIGKILL if needed user_scripts_stop.sh --dry-run # show which scripts would be stopped, by name user_scripts_stop.sh --status # show running scripts with PID and runtime user_scripts_stop.sh --log # verbose per-process output ``` --- ## ━━━ STARTUP SEQUENCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ```bash # master.conf — ARRAY_START_SCRIPTS (order matters) # ───────────────────────────────────────────────────────────────────────────── ARRAY_START_SCRIPTS=( # ── One-shot — run and exit ─────────────────────────────────────────────── "unRAID_Essentials/inotify_tuning.sh" # FIRST — raise limits before # containers inherit old values "unRAID_Essentials/docker_syslog_filter.sh" # SECOND — before containers # create veth interfaces "unRAID_Essentials/php_fpm_max_children.sh" # before WebGUI serves requests "Transcodes/ramdisk_setup.sh" # before Emby starts transcoding "Docker_Essentials/docker_network_connect.sh" # before watchdogs check states # ── Continuous — run until array stops ─────────────────────────────────── "unRAID_Essentials/system_watchdog.sh" # before docker_watchdog — # writes state file docker_watchdog reads "Docker_Essentials/docker_watchdog.sh" # before failover — containers # must be healthy for failover decisions "Failover/failover.sh" # last — needs everything stable ) # ───────────────────────────────────────────────────────────────────────────── # array_start.sh is the ONLY "At Startup of Array" entry in User Scripts. # It launches everything above in order. ``` --- ## ━━━ FULL SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ```bash # At Startup of Array — via array_start.sh: # inotify_tuning.sh # docker_syslog_filter.sh # php_fpm_max_children.sh # (ramdisk_setup.sh — in Transcodes/) # system_watchdog.sh (continuous) # Every 10 minutes: */10 * * * * webgui_restart.sh # silent when healthy — escalates when not # Weekly — via weekly_sync_maintenance.sh: # clear_logs.sh # Sunday 2:30am via WEEKLY_MAINTENANCE_SCRIPTS # Manual: # mover_stop.sh — before array ops that need mover stopped # server_reboot.sh — planned maintenance reboots # user_scripts_stop.sh — emergency script stop or pre-reboot cleanup ```