feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -2,71 +2,107 @@
|
||||
# ==============================================================================================
|
||||
# ================================= System Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Last line of defense — reboots the system cleanly if it is about to become unstable.
|
||||
# Runs continuously as a background process — started by array_started.sh at array start.
|
||||
# Works alongside docker_watchdog.sh which handles container-level healing first.
|
||||
#
|
||||
# ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Last line of defense — reboots the system cleanly if it is about to become
|
||||
# unstable. Runs continuously as a background process started by
|
||||
# array_started.sh at array start. Works alongside docker_watchdog.sh which
|
||||
# handles container-level healing first. Only escalates to reboot when
|
||||
# docker_watchdog.sh cannot resolve the condition.
|
||||
#
|
||||
# TIER 1 — CRITICAL (bypass ALL strikes, reboot immediately)
|
||||
# Docker daemon unresponsive — nothing can be healed, letting it run makes it worse
|
||||
# rootfs at 99%+ — writes failing, SSH may stop, no recovery options
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three-Tier Response System
|
||||
#
|
||||
# Tier 1 — CRITICAL (bypass all strikes, reboot immediately)
|
||||
# Docker daemon unresponsive — nothing can be healed; running it longer makes it worse
|
||||
# rootfs at 99%+ — writes failing; SSH may stop; no recovery options
|
||||
# Kernel oops/BUG in dmesg — kernel running with corrupted state
|
||||
# File descriptor exhaustion — new connections and processes failing silently
|
||||
# /boot read-only unexpectedly — state files and config writes silently failing
|
||||
#
|
||||
# TIER 2 — URGENT (bypass strikes when OOM confirms active crisis)
|
||||
# RAM < MEM_GB AND OOM kills >= OOM_LIMIT in this cycle
|
||||
# Rationale: OOM kills at this rate means system is dying faster than watchdogs heal
|
||||
# Without OOM confirmation → standard strike system applies
|
||||
# Tier 2 — URGENT (bypass strikes when OOM confirms active crisis)
|
||||
# RAM < MEM_GB AND OOM kills >= OOM_LIMIT in this cycle.
|
||||
# OOM kills at this rate means the system is dying faster than watchdogs can heal.
|
||||
# Without OOM confirmation → standard strike system applies.
|
||||
#
|
||||
# TIER 3 — STANDARD (N consecutive failures → reboot)
|
||||
# RAM tiers, load, CPU temp, zombies, /var/log, /tmp, containers, NIC, mdstat
|
||||
# Tier 3 — STANDARD (N consecutive failures → reboot)
|
||||
# RAM tiers, load, CPU temp, zombies, /var/log, /tmp, containers, NIC, mdstat.
|
||||
#
|
||||
# ── RAM TIERS ─────────────────────────────────────────────────────────────────────────────────
|
||||
# RAM Tiers
|
||||
# MEM_WARN_GB (10GB) — warn + notify only
|
||||
# MEM_SHUTDOWN_GB (6GB) — stop non-essential containers, wait for recovery
|
||||
# MEM_GB (4GB) — strike system → reboot (bypass if OOM confirms)
|
||||
# MEM_RECOVER_GB (30GB) — RAM must reach this before containers restart
|
||||
# MEM_GB (4GB) — strike system → reboot (bypass with OOM confirmation)
|
||||
# MEM_RECOVER_GB (30GB) — RAM must reach this before stopped containers restart
|
||||
#
|
||||
# ── CONTAINER SHUTDOWN LOGIC ──────────────────────────────────────────────────────────────────
|
||||
# At MEM_SHUTDOWN_GB: stop all containers NOT in SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED
|
||||
# Excluded: NginxProxyManager, Authelia, Mariadb, Redis, Emby, Dispatcharr
|
||||
# Stopped containers tracked in shutdown list — won't restart until RAM recovers
|
||||
# Strike system prevents flip-flopping — shutdown only happens once per degradation event
|
||||
# Container Shutdown Logic (at MEM_SHUTDOWN_GB)
|
||||
# Stops all containers not in SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED.
|
||||
# Stopped containers tracked in shutdown list — won't restart until RAM recovers.
|
||||
# Strike system prevents flip-flopping — shutdown only once per degradation event.
|
||||
#
|
||||
# ── OOM TRACKING ──────────────────────────────────────────────────────────────────────────────
|
||||
# /proc/vmstat oom_kill counter — read each cycle, delta = kills this cycle
|
||||
# Included in reboot message with process names from dmesg (diagnostic context)
|
||||
# Bypass trigger: RAM critical AND kills this cycle >= SYS_WATCHDOG_OOM_LIMIT
|
||||
# Abort Conditions (prevent reboot during sensitive operations)
|
||||
# ZFS pool unhealthy, parity running, mover running — each toggleable.
|
||||
# CRITICAL tier bypasses all abort conditions — imminent crash overrides data safety.
|
||||
#
|
||||
# ── NEW CHECKS THIS VERSION ───────────────────────────────────────────────────────────────────
|
||||
# OOM rate tracking — delta from /proc/vmstat each cycle
|
||||
# /boot read-only — write test on /boot each cycle
|
||||
# Kernel oops detection — dmesg BUG/Oops count delta each cycle
|
||||
# File descriptor exhaustion — /proc/sys/fs/file-nr utilisation
|
||||
# /tmp usage — tmpfs fill detection with auto-clear attempt
|
||||
# Array disk errors — mdstat error delta each cycle
|
||||
# Runaway process — single process >N% CPU sustained (disabled by default)
|
||||
# NIC state check — primary interface operstate
|
||||
# sshd check — restart attempt before escalating
|
||||
# Checks Run Every Cycle
|
||||
# rootfs usage, /var/log, /tmp, free RAM, ZFS ARC, CPU temp, load avg,
|
||||
# zombie processes, Docker daemon, OOM rate, /boot read-only, kernel oops,
|
||||
# file descriptor exhaustion, array disk errors, NIC state, required containers.
|
||||
#
|
||||
# ── EXISTING CHECKS ───────────────────────────────────────────────────────────────────────────
|
||||
# rootfs usage, /var/log, free RAM, ZFS ARC, CPU temp, load avg,
|
||||
# zombie processes, Docker daemon, required containers from skip list
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── ABORT CONDITIONS ──────────────────────────────────────────────────────────────────────────
|
||||
# ZFS pool unhealthy, parity running, mover running — toggleable
|
||||
# CRITICAL tier bypasses abort conditions — imminent crash overrides data safety
|
||||
# Root Required
|
||||
# Reboot and container stop require root.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# Full config under System Watchdog section — see master.conf for all vars
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents a second watchdog instance from starting.
|
||||
#
|
||||
# State File Verification
|
||||
# All state files verified writable at startup — errors if any cannot be created.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf — System Watchdog section
|
||||
# Full variable listing in master.conf. Key variables:
|
||||
#
|
||||
# SYS_WATCHDOG_REBOOT_WINDOW_HRS — reboot rate limit window (default: 2)
|
||||
# SYS_WATCHDOG_MAX_REBOOTS — max reboots in window before giving up (default: 3)
|
||||
# SYS_WATCHDOG_STRIKES — consecutive failures before reboot (default: 3)
|
||||
# SYS_WATCHDOG_OOM_LIMIT — OOM kills/cycle to trigger URGENT bypass (default: 3)
|
||||
# SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED — containers exempt from memory shutdown
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# SYS_WATCHDOG_STATE_FILE — strike counters and cycle state
|
||||
# SYS_WATCHDOG_REBOOT_LOG — reboot history for rate limiting
|
||||
# SYS_WATCHDOG_FAILED_FILE — containers confirmed down for skip list integration
|
||||
# SYS_WATCHDOG_OOM_FILE — OOM kill counter from previous cycle
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# system_watchdog.sh
|
||||
# Start continuous monitoring loop. Runs until stopped or system reboots.
|
||||
#
|
||||
# system_watchdog.sh --dry-run
|
||||
# Run detection logic without rebooting or stopping containers.
|
||||
#
|
||||
# system_watchdog.sh --status
|
||||
# Show config, thresholds, current system state, and strike counts.
|
||||
#
|
||||
# system_watchdog.sh --log
|
||||
# Verbose per-cycle output — show every check result and threshold comparison.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# system_watchdog.sh — normal start (continuous loop)
|
||||
# system_watchdog.sh --dry-run — trigger detection without rebooting
|
||||
# system_watchdog.sh --status — show config and thresholds
|
||||
# system_watchdog.sh --log — verbose per-cycle output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user