feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -2,81 +2,217 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Two-tier self-healing container monitoring system.
|
||||
# Runs continuously as a background process — started by array_started.sh at array start.
|
||||
# Shuts down cleanly on SIGTERM/SIGINT when array stops.
|
||||
#
|
||||
# ── TIER 1 — STRICT MONITORING ────────────────────────────────────────────────────────────────
|
||||
# Applies only to explicitly configured containers (HOST*_WATCHDOG_CONTAINERS etc.)
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Two-tier self-healing container monitoring system. Runs as a continuous
|
||||
# background daemon started by array_started.sh at array start. Shuts down
|
||||
# cleanly on SIGTERM/SIGINT when the array stops.
|
||||
#
|
||||
# Memory hard limits — immediate restart if container exceeds configured MB ceiling
|
||||
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of hard limit (no restart)
|
||||
# Every DOCKER_WATCHDOG_INTERVAL seconds the watchdog runs a full cycle:
|
||||
# Tier 1 applies specific thresholds to explicitly configured containers.
|
||||
# Tier 2 scans everything else for generic health problems. Silent on clean
|
||||
# cycles, loud when something needs attention.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tier 1 — Strict Per-Container Monitoring
|
||||
# Applies only to containers explicitly configured in master_host*.conf.
|
||||
#
|
||||
# Memory hard limits — immediate restart if container exceeds MB ceiling
|
||||
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart)
|
||||
# CPU thresholds — strike system: warn at SOFT_CPU_THRESHOLD, restart after
|
||||
# CPU_FAIL_LIMIT consecutive strikes at HARD_CPU_THRESHOLD
|
||||
# HTTP responsiveness — strike system: restart after RESP_FAIL_LIMIT consecutive failures
|
||||
# HTTP responsiveness — strike system: restart after RESP_FAIL_LIMIT consecutive
|
||||
# failures against the configured endpoint
|
||||
# Required containers — must always be running; strike system before restart;
|
||||
# skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window;
|
||||
# auto-clears when container recovers
|
||||
# skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window
|
||||
#
|
||||
# ── TIER 2 — GLOBAL HEALTH SCAN ───────────────────────────────────────────────────────────────
|
||||
# Scans ALL running containers when WATCHDOG_SCAN_ALL=true.
|
||||
# Containers in WATCHDOG_SCAN_IGNORE are excluded from Tier 2.
|
||||
# Tier 2 — Global Health Scan
|
||||
# Scans ALL running containers when WATCHDOG_SCAN_ALL=true.
|
||||
# Containers in WATCHDOG_SCAN_IGNORE are excluded.
|
||||
#
|
||||
# Unhealthy status — Docker HEALTHCHECK unhealthy → safe_restart()
|
||||
# OOM killed — kernel OOM killed → safe_restart() + notify
|
||||
# OOM state tracked per-session to prevent restart loop
|
||||
# Crash loop detection — RestartCount climbing → notify; above WATCHDOG_CRASH_LIMIT
|
||||
# → safe_restart() → skip list if restart limit hit
|
||||
# Dead containers — safe_restart() via remove + start
|
||||
# Unexpected exits — non-zero exit code → safe_restart()
|
||||
# Unhealthy status — Docker HEALTHCHECK unhealthy → restart
|
||||
# OOM killed — kernel OOM kill detected → restart + notify
|
||||
# Crash loop — RestartCount climbing → notify; above WATCHDOG_CRASH_LIMIT
|
||||
# → restart → skip list if restart limit hit
|
||||
# Dead containers — remove + start (dead state cannot be restarted directly)
|
||||
# Unexpected exits — non-zero exit code → restart
|
||||
#
|
||||
# Cross-Cutting Intelligence
|
||||
# Applies to both tiers on every cycle.
|
||||
#
|
||||
# ── CROSS-CUTTING INTELLIGENCE ────────────────────────────────────────────────────────────────
|
||||
# Startup grace period — no restarts for WATCHDOG_STARTUP_GRACE seconds after boot
|
||||
# Dependency ordering — waits for dependencies before restarting a dependent container
|
||||
# Restart loop protect — skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in rolling window
|
||||
# Skip list auto-clear — clears when container is seen running again
|
||||
# Dependency ordering — dependency restarted first, dependent skipped this cycle
|
||||
# Restart loop protect — skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window
|
||||
# Skip list auto-clear — removed when container seen running again
|
||||
# Notification batching — one summary per cycle, not one ping per event
|
||||
# Parity awareness — skips restart actions during parity check
|
||||
# Timeout protection — all docker commands wrapped in timeout — daemon hangs cannot
|
||||
# stall the watchdog and leave containers unmonitored
|
||||
# Docker daemon check — first check every cycle; hung daemon → strike system →
|
||||
# restart daemon via /etc/rc.d/rc.docker → verify recovery
|
||||
# system_watchdog.sh handles escalation if restart fails
|
||||
# Quiet when healthy — only logs when something needs attention (plus heartbeat)
|
||||
# Timeout protection — all docker commands wrapped in timeout
|
||||
# Docker daemon check — each cycle begins with daemon health check; hung daemon →
|
||||
# restart via rc.docker → system_watchdog.sh escalates if needed
|
||||
# RAM emergency defer — reads SYS_WATCHDOG_STATE_FILE; stands down while
|
||||
# system_watchdog.sh is managing a RAM emergency
|
||||
#
|
||||
# ── STATE FILES ───────────────────────────────────────────────────────────────────────────────
|
||||
# WATCHDOG_STATE_FILE — strike counts (/tmp — resets on reboot, correct)
|
||||
# SYS_WATCHDOG_FAILED_FILE — skip list (/boot — survives reboots, intentional)
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tiered Monitoring
|
||||
# Not all containers need the same monitoring strategy. Tier 1 gives explicit
|
||||
# control over the containers that matter most. Tier 2 is the catch-all that
|
||||
# requires no configuration and protects everything else.
|
||||
#
|
||||
# Strike vs Immediate
|
||||
# CPU spikes and HTTP failures are transient — brief spikes are normal during
|
||||
# transcoding or library scans. Memory leaks are not transient. CPU and HTTP
|
||||
# use a strike system to distinguish sustained problems from momentary ones.
|
||||
# Memory triggers immediate restart because a container at its ceiling is
|
||||
# actively leaking, not spiking.
|
||||
#
|
||||
# Loop Protection Over Persistence
|
||||
# A watchdog that keeps restarting a broken container is not helpful — it risks
|
||||
# making a database corruption worse. After WATCHDOG_CONTAINER_RESTART_LIMIT
|
||||
# attempts the container is skip-listed and the operator is notified. Automated
|
||||
# recovery stops. Human investigation begins.
|
||||
#
|
||||
# Dependency-Safe Ordering
|
||||
# When a container and its dependency are both down, restart the dependency
|
||||
# first and skip the dependent this cycle. Prevents false-alarm skip-listing
|
||||
# of containers whose only failure was starting before their dependency was ready.
|
||||
#
|
||||
# Silent When Healthy
|
||||
# Runs 96 times per day. Producing output on every clean cycle would make
|
||||
# logs useless. Output only when something needs attention or a heartbeat fires.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Startup Grace Period
|
||||
# Restart actions suppressed for WATCHDOG_STARTUP_GRACE seconds after the
|
||||
# watchdog starts. Checks still run and log — only restarts are suppressed.
|
||||
# Prevents false-positive restarts while containers are still initialising.
|
||||
#
|
||||
# Restart Loop Protection
|
||||
# WATCHDOG_CONTAINER_RESTART_LIMIT restarts within WATCHDOG_CONTAINER_RESTART_WINDOW
|
||||
# hours triggers skip-listing and a critical notification. Skip list persists on
|
||||
# /boot/config/ — survives reboots intentionally. Auto-clears when container
|
||||
# is seen running again.
|
||||
#
|
||||
# Docker Daemon Health Check
|
||||
# First operation every cycle. Daemon not responding within DOCKER_TIMEOUT →
|
||||
# restart via /etc/rc.d/rc.docker → verify recovery. If still hung: log
|
||||
# critical, skip cycle. system_watchdog.sh handles further escalation.
|
||||
#
|
||||
# RAM Emergency Deferral
|
||||
# Reads SYS_WATCHDOG_STATE_FILE each cycle. If system_watchdog.sh has set
|
||||
# mem_shutdown_active=true, all restart logic defers until the flag clears.
|
||||
# Stale state guard: if file is >2 hours old with flag still set,
|
||||
# system_watchdog.sh has likely stopped — watchdog resumes normal operation.
|
||||
#
|
||||
# Timeout Protection
|
||||
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
|
||||
# watchdog and leave containers unmonitored between cycles.
|
||||
#
|
||||
# Notification Batching
|
||||
# Events collected across a full cycle and sent as a single summary.
|
||||
# Prevents notification floods when a shared dependency failure cascades.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# WATCHDOG_STATE_FILE — strike counts (default: /tmp — resets on reboot)
|
||||
# SYS_WATCHDOG_FAILED_FILE — skip list (default: /boot/config — survives reboots)
|
||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection
|
||||
# SYS_WATCHDOG_STATE_FILE — shared state with system_watchdog.sh (RAM emergency flag)
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_WATCHDOG_CONTAINERS — memory hard limits per container
|
||||
# HOST*_WATCHDOG_CONTAINER_URLS — HTTP health check URLs
|
||||
# HOST*_WATCHDOG_REQUIRED_CONTAINERS — must always be running
|
||||
# HOST*_WATCHDOG_SCAN_IGNORE — skip in Tier 2 scan
|
||||
# HOST*_WATCHDOG_DEPENDENCIES — dependency ordering for restart decisions
|
||||
# All aliased by detect_hosts() — script uses unprefixed names
|
||||
# /tmp files reset on reboot — correct, pre-reboot strike counts are meaningless after it.
|
||||
# /boot/config files survive reboots — correct, a skip-listed container is still broken after one.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_WATCHDOG_CONTAINERS
|
||||
# Memory hard limits per container. Format: "ContainerName:LimitInMB"
|
||||
# Aliased by detect_hosts() → WATCHDOG_CONTAINERS
|
||||
#
|
||||
# HOST*_WATCHDOG_CONTAINER_URLS
|
||||
# HTTP health check endpoints. Format: "ContainerName:http://host:port"
|
||||
# Aliased by detect_hosts() → WATCHDOG_CONTAINER_URLS
|
||||
#
|
||||
# HOST*_WATCHDOG_REQUIRED_CONTAINERS
|
||||
# Containers that must always be running. Aliased by detect_hosts() →
|
||||
# WATCHDOG_REQUIRED_CONTAINERS
|
||||
#
|
||||
# HOST*_WATCHDOG_SCAN_IGNORE
|
||||
# Containers excluded from Tier 2 global scan. Aliased by detect_hosts() →
|
||||
# WATCHDOG_SCAN_IGNORE
|
||||
#
|
||||
# HOST*_WATCHDOG_DEPENDENCIES
|
||||
# Dependency ordering. Format: "Dependent:dep1 dep2". Aliased by
|
||||
# detect_hosts() → WATCHDOG_DEPENDENCIES
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# SOFT_CPU_THRESHOLD / HARD_CPU_THRESHOLD / CPU_FAIL_LIMIT
|
||||
# SOFT_MEM_THRESHOLD
|
||||
# RESP_FAIL_LIMIT / CURL_TIMEOUT
|
||||
# DOCKER_WATCHDOG_INTERVAL
|
||||
# DOCKER_WATCHDOG_HEARTBEAT / DOCKER_WATCHDOG_HEARTBEAT_HOURS
|
||||
# Seconds between full watchdog cycles (default: 900)
|
||||
#
|
||||
# WATCHDOG_STARTUP_GRACE
|
||||
# Seconds before restart actions begin after watchdog starts (default: 600)
|
||||
#
|
||||
# SOFT_MEM_THRESHOLD
|
||||
# Warn at this % of hard memory limit — no restart (default: 80)
|
||||
#
|
||||
# SOFT_CPU_THRESHOLD / HARD_CPU_THRESHOLD / CPU_FAIL_LIMIT
|
||||
# CPU monitoring thresholds and strike limit
|
||||
#
|
||||
# CURL_TIMEOUT / RESP_FAIL_LIMIT
|
||||
# HTTP health check timeout and consecutive failure limit
|
||||
#
|
||||
# WATCHDOG_SCAN_ALL
|
||||
# Enable Tier 2 global health scan (default: true)
|
||||
#
|
||||
# WATCHDOG_RESTART_UNHEALTHY / WATCHDOG_RESTART_DEAD / WATCHDOG_RESTART_CRASHED
|
||||
# WATCHDOG_NOTIFY_OOM / WATCHDOG_NOTIFY_CRASHLOOP
|
||||
# WATCHDOG_CRASH_LIMIT
|
||||
# WATCHDOG_STARTUP_GRACE
|
||||
# Tier 2 action toggles
|
||||
#
|
||||
# WATCHDOG_NOTIFY_OOM / WATCHDOG_NOTIFY_CRASHLOOP / WATCHDOG_CRASH_LIMIT
|
||||
# OOM and crash loop detection toggles and threshold
|
||||
#
|
||||
# WATCHDOG_CONTAINER_RESTART_LIMIT / WATCHDOG_CONTAINER_RESTART_WINDOW
|
||||
# Restart loop protection: attempt limit and rolling window in hours
|
||||
#
|
||||
# WATCHDOG_BATCH_NOTIFY
|
||||
# WATCHDOG_STATE_FILE / SYS_WATCHDOG_FAILED_FILE / WATCHDOG_CONTAINER_RESTART_LOG
|
||||
# Collect cycle events and send as one notification (default: true)
|
||||
#
|
||||
# DOCKER_WATCHDOG_HEARTBEAT_HOURS
|
||||
# Hours between alive heartbeat log entries
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_watchdog.sh
|
||||
# Start continuous monitoring loop — normally launched by array_started.sh
|
||||
#
|
||||
# docker_watchdog.sh --dry-run
|
||||
# Run a full watchdog cycle without restarting anything. Shows what would
|
||||
# happen based on current container states. Use to verify configuration.
|
||||
#
|
||||
# docker_watchdog.sh --status
|
||||
# Show strike counts, skip list contents, grace period status, RAM emergency
|
||||
# deferral state, and last cycle timing. Then exit.
|
||||
#
|
||||
# docker_watchdog.sh --log
|
||||
# Verbose output — full detail for every container checked and every decision.
|
||||
# Use to debug why a container is or is not being restarted.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# docker_watchdog.sh — normal start (continuous loop)
|
||||
# docker_watchdog.sh --dry-run — preview without restarting
|
||||
# docker_watchdog.sh --status — show config and exit
|
||||
# docker_watchdog.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user