Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder: Docker_Essentials/docker_watchdog.sh → Watchdogs/ unRAID_Essentials/system_watchdog.sh → Watchdogs/ unRAID_Essentials/resource_watchdog.sh → Watchdogs/ Orchestrators/watchdog_orchestrator.sh → Watchdogs/ Tools/watchdog_skip_list_manager.sh → Watchdogs/ Rename host config files: master_host1.conf → host1.conf master_host2.conf → host2.conf Update all references across the ecosystem: master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/ load_config.sh: host*.conf glob + all comments git_pull_execute.sh: sparse checkout glob + all comments Partnership/ssh_setup.sh: HOST_CONF path construction user_script_plug-in.sh: all script paths + per-host conf path common.sh, README.md, README-User_Script_Plug-in.md: comment refs All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
This commit is contained in:
Gmer4Lfe
Executable
+943
@@ -0,0 +1,943 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Two-tier self-healing container monitoring system. Runs as a continuous
|
||||
# background daemon started by array_started.sh at array start. Shuts down
|
||||
# cleanly on SIGTERM/SIGINT when the array stops.
|
||||
#
|
||||
# Every DOCKER_WATCHDOG_INTERVAL seconds the watchdog runs a full cycle:
|
||||
# Tier 1 applies specific thresholds to explicitly configured containers.
|
||||
# Tier 2 scans everything else for generic health problems. Silent on clean
|
||||
# cycles, loud when something needs attention.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tier 1 — Strict Per-Container Monitoring
|
||||
# Applies only to containers explicitly configured in host*.conf.
|
||||
#
|
||||
# Memory hard limits — immediate restart if container exceeds MB ceiling
|
||||
# Memory soft threshold — warn at SOFT_MEM_THRESHOLD % of limit (no restart)
|
||||
# CPU thresholds — strike system: warn at SOFT_CPU_THRESHOLD, restart after
|
||||
# CPU_FAIL_LIMIT consecutive strikes at HARD_CPU_THRESHOLD
|
||||
# HTTP responsiveness — strike system: restart after RESP_FAIL_LIMIT consecutive
|
||||
# failures against the configured endpoint
|
||||
# Required containers — must always be running; strike system before restart;
|
||||
# skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window
|
||||
#
|
||||
# Tier 2 — Global Health Scan
|
||||
# Scans ALL running containers when WATCHDOG_SCAN_ALL=true.
|
||||
# Containers in WATCHDOG_SCAN_IGNORE are excluded.
|
||||
#
|
||||
# Unhealthy status — Docker HEALTHCHECK unhealthy → restart
|
||||
# OOM killed — kernel OOM kill detected → restart + notify
|
||||
# Crash loop — RestartCount climbing → notify; above WATCHDOG_CRASH_LIMIT
|
||||
# → restart → skip list if restart limit hit
|
||||
# Dead containers — remove + start (dead state cannot be restarted directly)
|
||||
# Unexpected exits — non-zero exit code → restart
|
||||
#
|
||||
# Cross-Cutting Intelligence
|
||||
# Applies to both tiers on every cycle.
|
||||
#
|
||||
# Startup grace period — no restarts for WATCHDOG_STARTUP_GRACE seconds after boot
|
||||
# Dependency ordering — dependency restarted first, dependent skipped this cycle
|
||||
# Restart loop protect — skip list after WATCHDOG_CONTAINER_RESTART_LIMIT in window
|
||||
# Skip list auto-clear — removed when container seen running again
|
||||
# Notification batching — one summary per cycle, not one ping per event
|
||||
# Parity awareness — skips restart actions during parity check
|
||||
# Timeout protection — all docker commands wrapped in timeout
|
||||
# Docker daemon check — each cycle begins with daemon health check; hung daemon →
|
||||
# restart via rc.docker → system_watchdog.sh escalates if needed
|
||||
# RAM emergency defer — reads SYS_WATCHDOG_STATE_FILE; stands down while
|
||||
# system_watchdog.sh is managing a RAM emergency
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tiered Monitoring
|
||||
# Not all containers need the same monitoring strategy. Tier 1 gives explicit
|
||||
# control over the containers that matter most. Tier 2 is the catch-all that
|
||||
# requires no configuration and protects everything else.
|
||||
#
|
||||
# Strike vs Immediate
|
||||
# CPU spikes and HTTP failures are transient — brief spikes are normal during
|
||||
# transcoding or library scans. Memory leaks are not transient. CPU and HTTP
|
||||
# use a strike system to distinguish sustained problems from momentary ones.
|
||||
# Memory triggers immediate restart because a container at its ceiling is
|
||||
# actively leaking, not spiking.
|
||||
#
|
||||
# Loop Protection Over Persistence
|
||||
# A watchdog that keeps restarting a broken container is not helpful — it risks
|
||||
# making a database corruption worse. After WATCHDOG_CONTAINER_RESTART_LIMIT
|
||||
# attempts the container is skip-listed and the operator is notified. Automated
|
||||
# recovery stops. Human investigation begins.
|
||||
#
|
||||
# Dependency-Safe Ordering
|
||||
# When a container and its dependency are both down, restart the dependency
|
||||
# first and skip the dependent this cycle. Prevents false-alarm skip-listing
|
||||
# of containers whose only failure was starting before their dependency was ready.
|
||||
#
|
||||
# Silent When Healthy
|
||||
# Runs 96 times per day. Producing output on every clean cycle would make
|
||||
# logs useless. Output only when something needs attention or a heartbeat fires.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe at array start —
|
||||
# only one watchdog instance runs at a time.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# all HOST*_WATCHDOG_* arrays to the correct host's values.
|
||||
#
|
||||
# Startup Grace Period
|
||||
# Restart actions suppressed for WATCHDOG_STARTUP_GRACE seconds after the
|
||||
# watchdog starts. Checks still run and log — only restarts are suppressed.
|
||||
# Prevents false-positive restarts while containers are still initialising.
|
||||
#
|
||||
# Restart Loop Protection
|
||||
# WATCHDOG_CONTAINER_RESTART_LIMIT restarts within WATCHDOG_CONTAINER_RESTART_WINDOW
|
||||
# hours triggers skip-listing and a critical notification. Skip list persists on
|
||||
# /boot/config/ — survives reboots intentionally. Auto-clears when container
|
||||
# is seen running again.
|
||||
#
|
||||
# Docker Daemon Health Check
|
||||
# First operation every cycle. Daemon not responding within DOCKER_TIMEOUT →
|
||||
# restart via /etc/rc.d/rc.docker → verify recovery. If still hung: log
|
||||
# critical, skip cycle. system_watchdog.sh handles further escalation.
|
||||
#
|
||||
# RAM Emergency Deferral
|
||||
# Reads SYS_WATCHDOG_STATE_FILE each cycle. If system_watchdog.sh has set
|
||||
# mem_shutdown_active=true, all restart logic defers until the flag clears.
|
||||
# Stale state guard: if file is >2 hours old with flag still set,
|
||||
# system_watchdog.sh has likely stopped — watchdog resumes normal operation.
|
||||
#
|
||||
# Timeout Protection
|
||||
# All docker commands wrapped in timeout. Daemon hangs cannot stall the
|
||||
# watchdog and leave containers unmonitored between cycles.
|
||||
#
|
||||
# Notification Batching
|
||||
# Events collected across a full cycle and sent as a single summary.
|
||||
# Prevents notification floods when a shared dependency failure cascades.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# WATCHDOG_STATE_FILE — strike counts (default: /tmp — resets on reboot)
|
||||
# SYS_WATCHDOG_FAILED_FILE — skip list (default: /boot/config — survives reboots)
|
||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection
|
||||
# SYS_WATCHDOG_STATE_FILE — shared state with system_watchdog.sh (RAM emergency flag)
|
||||
#
|
||||
# /tmp files reset on reboot — correct, pre-reboot strike counts are meaningless after it.
|
||||
# /boot/config files survive reboots — correct, a skip-listed container is still broken after one.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_WATCHDOG_CONTAINERS
|
||||
# Memory hard limits per container. Format: "ContainerName:LimitInMB"
|
||||
# Aliased by detect_hosts() → WATCHDOG_CONTAINERS
|
||||
#
|
||||
# HOST*_WATCHDOG_CONTAINER_URLS
|
||||
# HTTP health check endpoints. Format: "ContainerName:http://host:port"
|
||||
# Aliased by detect_hosts() → WATCHDOG_CONTAINER_URLS
|
||||
#
|
||||
# HOST*_WATCHDOG_REQUIRED_CONTAINERS
|
||||
# Containers that must always be running. Aliased by detect_hosts() →
|
||||
# WATCHDOG_REQUIRED_CONTAINERS
|
||||
#
|
||||
# HOST*_WATCHDOG_SCAN_IGNORE
|
||||
# Containers excluded from Tier 2 global scan. Aliased by detect_hosts() →
|
||||
# WATCHDOG_SCAN_IGNORE
|
||||
#
|
||||
# HOST*_WATCHDOG_DEPENDENCIES
|
||||
# Dependency ordering. Format: "Dependent:dep1 dep2". Aliased by
|
||||
# detect_hosts() → WATCHDOG_DEPENDENCIES
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DOCKER_WATCHDOG_INTERVAL
|
||||
# Seconds between full watchdog cycles (default: 900)
|
||||
#
|
||||
# WATCHDOG_STARTUP_GRACE
|
||||
# Seconds before restart actions begin after watchdog starts (default: 600)
|
||||
#
|
||||
# SOFT_MEM_THRESHOLD
|
||||
# Warn at this % of hard memory limit — no restart (default: 80)
|
||||
#
|
||||
# SOFT_CPU_THRESHOLD / HARD_CPU_THRESHOLD / CPU_FAIL_LIMIT
|
||||
# CPU monitoring thresholds and strike limit
|
||||
#
|
||||
# CURL_TIMEOUT / RESP_FAIL_LIMIT
|
||||
# HTTP health check timeout and consecutive failure limit
|
||||
#
|
||||
# WATCHDOG_SCAN_ALL
|
||||
# Enable Tier 2 global health scan (default: true)
|
||||
#
|
||||
# WATCHDOG_RESTART_UNHEALTHY / WATCHDOG_RESTART_DEAD / WATCHDOG_RESTART_CRASHED
|
||||
# Tier 2 action toggles
|
||||
#
|
||||
# WATCHDOG_NOTIFY_OOM / WATCHDOG_NOTIFY_CRASHLOOP / WATCHDOG_CRASH_LIMIT
|
||||
# OOM and crash loop detection toggles and threshold
|
||||
#
|
||||
# WATCHDOG_CONTAINER_RESTART_LIMIT / WATCHDOG_CONTAINER_RESTART_WINDOW
|
||||
# Restart loop protection: attempt limit and rolling window in hours
|
||||
#
|
||||
# WATCHDOG_BATCH_NOTIFY
|
||||
# Collect cycle events and send as one notification (default: true)
|
||||
#
|
||||
# DOCKER_WATCHDOG_HEARTBEAT_HOURS
|
||||
# Hours between alive heartbeat log entries
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_watchdog.sh
|
||||
# Start continuous monitoring loop — normally launched by array_started.sh
|
||||
#
|
||||
# docker_watchdog.sh --dry-run
|
||||
# Run a full watchdog cycle without restarting anything. Shows what would
|
||||
# happen based on current container states. Use to verify configuration.
|
||||
#
|
||||
# docker_watchdog.sh --status
|
||||
# Show strike counts, skip list contents, grace period status, RAM emergency
|
||||
# deferral state, and last cycle timing. Then exit.
|
||||
#
|
||||
# docker_watchdog.sh --log
|
||||
# Verbose output — full detail for every container checked and every decision.
|
||||
# Use to debug why a container is or is not being restarted.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup — runs once at start ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Skip if another instance is running — no pile-up during long operations
|
||||
acquire_lock
|
||||
|
||||
# detect_hosts() sets MY_ID and aliases all HOST*_WATCHDOG_* arrays
|
||||
detect_hosts
|
||||
|
||||
if ! command -v docker >/dev/null 2>&1; then
|
||||
error "Docker not found — cannot start watchdog"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be restarted"
|
||||
|
||||
# Validate unRAID-specific commands used by this script
|
||||
# If rc.docker is missing or changed, daemon restart will fail — better to know now
|
||||
validate_unraid_cmd "/etc/rc.d/rc.docker" "" "" "Docker rc.d script" || warn "rc.docker not found — daemon restart unavailable if needed"
|
||||
|
||||
validate_unraid_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
# Ensure state files exist
|
||||
touch "$WATCHDOG_STATE_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" \
|
||||
"$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null
|
||||
|
||||
# Timeout for all docker commands — prevents hung daemon from stalling the watchdog
|
||||
DOCKER_TIMEOUT=15
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
_watched="${!WATCHDOG_CONTAINERS[*]}"; echo "$ICON_CONTAINERS Watched: ${_watched:-none}"
|
||||
echo "$ICON_CONTAINERS Required: ${WATCHDOG_REQUIRED_CONTAINERS[*]:-none}"
|
||||
echo "$ICON_WATCHDOG Scan all: $WATCHDOG_SCAN_ALL"
|
||||
echo "$ICON_WATCHDOG Ignore: ${WATCHDOG_SCAN_IGNORE[*]:-none}"
|
||||
echo "$ICON_WATCHDOG Interval: ${DOCKER_WATCHDOG_INTERVAL}s"
|
||||
echo "$ICON_WATCHDOG Startup grace: ${WATCHDOG_STARTUP_GRACE}s"
|
||||
echo "$ICON_WATCHDOG Restart limit: $WATCHDOG_CONTAINER_RESTART_LIMIT in ${WATCHDOG_CONTAINER_RESTART_WINDOW}h"
|
||||
echo "$ICON_WATCHDOG Batch notify: $WATCHDOG_BATCH_NOTIFY"
|
||||
echo "$ICON_WATCHDOG Docker timeout: ${DOCKER_TIMEOUT}s"
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
UPTIME_S=$(awk '{print int($1)}' /proc/uptime)
|
||||
echo "$ICON_TIME System uptime: $(format_duration $UPTIME_S)"
|
||||
[[ "$UPTIME_S" -lt "$WATCHDOG_STARTUP_GRACE" ]] && \
|
||||
warn "Within startup grace period — restarts suppressed"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── HELPER FUNCTIONS ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# Get strike count for a container from state file
|
||||
get_strikes() {
|
||||
grep "^${1}:" "${2}" 2>/dev/null | cut -d: -f2 || echo "0"
|
||||
}
|
||||
|
||||
# Set strike count for a container in state file
|
||||
set_strikes() {
|
||||
local container="$1" count="$2" file="$3"
|
||||
if grep -q "^${container}:" "$file" 2>/dev/null; then
|
||||
sed -i "s/^${container}:.*/${container}:${count}/" "$file"
|
||||
else
|
||||
echo "${container}:${count}" >> "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
# Check if container is on the persistent skip list
|
||||
is_skipped() {
|
||||
grep -q "^${1}$" "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null
|
||||
}
|
||||
|
||||
# Add container to persistent skip list — manual intervention required to recover
|
||||
add_to_skip_list() {
|
||||
local container="$1" reason="$2"
|
||||
if ! is_skipped "$container"; then
|
||||
echo "$container" >> "$SYS_WATCHDOG_FAILED_FILE"
|
||||
error "$container added to skip list — $reason"
|
||||
queue_notify "$container added to skip list on $(hostname) — $reason — manual intervention needed" "critical"
|
||||
fi
|
||||
}
|
||||
|
||||
# Remove container from skip list — called when container is seen running again
|
||||
remove_from_skip_list() {
|
||||
sed -i "/^${1}$/d" "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null
|
||||
warn "$1 recovered — removed from skip list ✅"
|
||||
}
|
||||
|
||||
# Log a restart event to the rolling restart history file
|
||||
log_restart() {
|
||||
local container="$1"
|
||||
local now
|
||||
now=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
local cutoff
|
||||
cutoff=$(date -d "${WATCHDOG_CONTAINER_RESTART_WINDOW} hours ago" '+%Y-%m-%d %H:%M:%S')
|
||||
echo "${container}|${now}" >> "$WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
# Trim entries older than the rolling window
|
||||
local tmp="${WATCHDOG_CONTAINER_RESTART_LOG}.tmp"
|
||||
awk -F'|' -v cutoff="$cutoff" '$2 >= cutoff' \
|
||||
"$WATCHDOG_CONTAINER_RESTART_LOG" > "$tmp" && \
|
||||
mv "$tmp" "$WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
}
|
||||
|
||||
# Get number of times a container was restarted within the rolling window
|
||||
get_restart_count() {
|
||||
local container="$1"
|
||||
local cutoff
|
||||
cutoff=$(date -d "${WATCHDOG_CONTAINER_RESTART_WINDOW} hours ago" '+%Y-%m-%d %H:%M:%S')
|
||||
awk -F'|' -v c="$container" -v cutoff="$cutoff" \
|
||||
'$1==c && $2>=cutoff' "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null | wc -l
|
||||
}
|
||||
|
||||
# Check if all dependencies of a container are currently running.
|
||||
# Returns 0 if all deps running (or no deps), 1 if any dep is down.
|
||||
dependencies_satisfied() {
|
||||
local container="$1"
|
||||
local deps="${WATCHDOG_DEPENDENCIES[$container]:-}"
|
||||
[[ -z "$deps" ]] && return 0
|
||||
for dep in $deps; do
|
||||
local status
|
||||
status=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$dep" 2>/dev/null)
|
||||
if [[ "$status" != "true" ]]; then
|
||||
warn "$container — dependency $dep is not running — skipping restart this cycle"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
# Safe restart with all guards:
|
||||
# - Restart loop protection (skip list after limit)
|
||||
# - Dependency check (don't restart if deps down)
|
||||
# - Startup grace period (no restarts while booting)
|
||||
# - Dry run support
|
||||
# - Timeout protection on docker restart
|
||||
#
|
||||
# Returns: 0=restarted, 1=skipped, 2=added to skip list
|
||||
safe_restart() {
|
||||
local container="$1" reason="$2"
|
||||
|
||||
# Restart loop protection — skip list if over limit
|
||||
local restart_count
|
||||
restart_count=$(get_restart_count "$container")
|
||||
if [[ "$restart_count" -ge "$WATCHDOG_CONTAINER_RESTART_LIMIT" ]]; then
|
||||
add_to_skip_list "$container" \
|
||||
"restarted $restart_count times in ${WATCHDOG_CONTAINER_RESTART_WINDOW}h"
|
||||
return 2
|
||||
fi
|
||||
|
||||
# Dependency check
|
||||
dependencies_satisfied "$container" || return 1
|
||||
|
||||
# Startup grace period
|
||||
local uptime_s
|
||||
uptime_s=$(awk '{print int($1)}' /proc/uptime)
|
||||
if [[ "$uptime_s" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
|
||||
warn "$container — within startup grace period (${uptime_s}s < ${WATCHDOG_STARTUP_GRACE}s) — skipping"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $container ($reason)"
|
||||
return 0
|
||||
fi
|
||||
|
||||
log "Restarting $container ($reason) [restart $((restart_count + 1))/$WATCHDOG_CONTAINER_RESTART_LIMIT in window]"
|
||||
if timeout "$DOCKER_TIMEOUT" docker restart "$container" >/dev/null 2>&1; then
|
||||
success "$ICON_STARTED $container restarted"
|
||||
log_restart "$container"
|
||||
return 0
|
||||
else
|
||||
error "Failed to restart $container (timeout or error)"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# Queue a notification event for batch sending at end of cycle
|
||||
queue_notify() {
|
||||
local message="$1" severity="${2:-warning}"
|
||||
NOTIFY_EVENTS+=("${severity}|${message}")
|
||||
log "Queued: $message"
|
||||
}
|
||||
|
||||
# Send all queued notifications — one batched summary or individual per event
|
||||
flush_notify() {
|
||||
[[ ${#NOTIFY_EVENTS[@]} -eq 0 ]] && return
|
||||
if [[ "$WATCHDOG_BATCH_NOTIFY" == "true" ]]; then
|
||||
local highest_severity="normal"
|
||||
local messages=()
|
||||
for event in "${NOTIFY_EVENTS[@]}"; do
|
||||
local sev="${event%%|*}" msg="${event#*|}"
|
||||
messages+=("$msg")
|
||||
[[ "$sev" == "critical" ]] && highest_severity="warning"
|
||||
[[ "$sev" == "warning" && "$highest_severity" == "normal" ]] && highest_severity="warning"
|
||||
done
|
||||
local summary
|
||||
summary=$(printf '%s. ' "${messages[@]}")
|
||||
notify "Docker Watchdog on $(hostname) — ${#NOTIFY_EVENTS[@]} event(s): $summary" \
|
||||
"Docker Watchdog" "$highest_severity"
|
||||
else
|
||||
for event in "${NOTIFY_EVENTS[@]}"; do
|
||||
local sev="${event%%|*}" msg="${event#*|}"
|
||||
[[ "$sev" == "critical" ]] && sev="warning"
|
||||
notify "$msg" "Docker Watchdog" "$sev"
|
||||
done
|
||||
fi
|
||||
NOTIFY_EVENTS=()
|
||||
}
|
||||
|
||||
# Returns 0 if parity check is currently running
|
||||
is_parity_running() {
|
||||
grep -q "progress" /var/local/emhttp/parity-date.txt 2>/dev/null
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── DOCKER DAEMON HEALTH CHECK ────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Checks Docker daemon responsiveness at the start of every cycle.
|
||||
# A hung daemon makes all container operations useless — check first, short-circuit if down.
|
||||
#
|
||||
# Strike system:
|
||||
# Each consecutive failed check adds a strike
|
||||
# At WATCHDOG_DAEMON_STRIKE_LIMIT → attempt daemon restart via rc.docker
|
||||
# After restart → wait WATCHDOG_DAEMON_RESTART_WAIT seconds → verify
|
||||
# If verified → clear strikes, continue cycle ✅
|
||||
# If still hung → notify critical, skip cycle → system_watchdog.sh escalates from here
|
||||
#
|
||||
# Returns: 0 = daemon healthy | 1 = daemon down, skip this cycle
|
||||
|
||||
WATCHDOG_DAEMON_STRIKE_LIMIT=3 # consecutive failed checks before restart attempt
|
||||
WATCHDOG_DAEMON_RESTART_WAIT=30 # seconds to wait after restart before verifying
|
||||
# Loaded from state file — persists across single-pass runs
|
||||
WATCHDOG_DAEMON_STRIKES=$(get_strikes "daemon_strikes" "$WATCHDOG_STATE_FILE")
|
||||
WATCHDOG_DAEMON_STRIKES="${WATCHDOG_DAEMON_STRIKES//[^0-9]/}"; WATCHDOG_DAEMON_STRIKES="${WATCHDOG_DAEMON_STRIKES:-0}"
|
||||
_dr_raw=$(get_strikes "daemon_restarted_flag" "$WATCHDOG_STATE_FILE")
|
||||
[[ "$_dr_raw" == "true" ]] && WATCHDOG_DAEMON_RESTARTED=true || WATCHDOG_DAEMON_RESTARTED=false
|
||||
|
||||
# ==============================================================================================
|
||||
# ── SYSTEM WATCHDOG COORDINATION ──────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Reads system_watchdog.sh state file to check if a RAM emergency shutdown is active.
|
||||
# During RAM emergency: system_watchdog.sh has stopped non-essential containers to free RAM.
|
||||
# docker_watchdog.sh must not restart them — that would undo the emergency shutdown and
|
||||
# prevent RAM from recovering, creating an infinite restart/shutdown loop.
|
||||
#
|
||||
# Returns:
|
||||
# 0 = normal — run all checks
|
||||
# 1 = RAM emergency active — defer container management this cycle
|
||||
|
||||
check_system_watchdog_state() {
|
||||
# Returns 0 = normal operation | 1 = defer, resource_watchdog RAM emergency active
|
||||
local state_file="$RW_STATE_FILE"
|
||||
|
||||
# No state file = resource_watchdog not yet run — assume normal
|
||||
[[ ! -f "$state_file" ]] && return 0
|
||||
|
||||
local mem_shutdown
|
||||
mem_shutdown=$(grep "^mem_shutdown_active=" "$state_file" 2>/dev/null | cut -d= -f2)
|
||||
|
||||
[[ "$mem_shutdown" != "true" ]] && return 0
|
||||
|
||||
# ── Stale state guard ─────────────────────────────────────────────────────────────────────
|
||||
# If mem_shutdown_active=true but state file hasn't been updated in > 2 hours,
|
||||
# resource_watchdog.sh may not be running — don't defer indefinitely on stale state.
|
||||
local state_mtime now age_seconds stale_limit=7200 # 2 hours
|
||||
state_mtime=$(stat -c %Y "$state_file" 2>/dev/null || echo 0)
|
||||
now=$(date +%s)
|
||||
age_seconds=$(( now - state_mtime ))
|
||||
|
||||
if [[ "$age_seconds" -gt "$stale_limit" ]]; then
|
||||
warn "mem_shutdown_active=true but state file is ${age_seconds}s old — may be stale"
|
||||
warn "resource_watchdog.sh may not be running — resuming normal container management"
|
||||
return 0 # Resume normal — don't defer indefinitely on stale state
|
||||
fi
|
||||
|
||||
return 1 # Defer — RAM emergency confirmed and state is fresh
|
||||
}
|
||||
|
||||
check_docker_daemon() {
|
||||
# docker info is more definitive than docker ps for daemon health
|
||||
if timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
|
||||
if [[ "$WATCHDOG_DAEMON_STRIKES" -gt 0 ]]; then
|
||||
log "$ICON_STARTED Docker daemon recovered — clearing strikes"
|
||||
queue_notify "Docker daemon recovered on $(hostname)" "normal"
|
||||
WATCHDOG_DAEMON_STRIKES=0
|
||||
WATCHDOG_DAEMON_RESTARTED=false
|
||||
set_strikes "daemon_strikes" 0 "$WATCHDOG_STATE_FILE"
|
||||
set_strikes "daemon_restarted_flag" "false" "$WATCHDOG_STATE_FILE"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
|
||||
WATCHDOG_DAEMON_STRIKES=$(( WATCHDOG_DAEMON_STRIKES + 1 ))
|
||||
set_strikes "daemon_strikes" "$WATCHDOG_DAEMON_STRIKES" "$WATCHDOG_STATE_FILE"
|
||||
warn "$ICON_WATCHDOG Docker daemon not responding (strike $WATCHDOG_DAEMON_STRIKES/$WATCHDOG_DAEMON_STRIKE_LIMIT)"
|
||||
|
||||
if [[ "$WATCHDOG_DAEMON_STRIKES" -lt "$WATCHDOG_DAEMON_STRIKE_LIMIT" ]]; then
|
||||
warn "Skipping monitoring cycle — waiting for daemon to recover"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "$WATCHDOG_DAEMON_RESTARTED" == true ]]; then
|
||||
error "Docker daemon still unresponsive after restart attempt"
|
||||
error "system_watchdog.sh will handle further escalation"
|
||||
queue_notify "Docker daemon hung on $(hostname) — restart failed — manual intervention needed" "critical"
|
||||
flush_notify
|
||||
return 1
|
||||
fi
|
||||
|
||||
error "Docker daemon unresponsive — attempting restart"
|
||||
notify "Docker daemon hung on $(hostname) — attempting restart" "Docker Watchdog" "warning"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart Docker daemon via /etc/rc.d/rc.docker restart"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Restart daemon — unRAID uses rc.d scripts, not systemd
|
||||
WATCHDOG_DAEMON_RESTARTED=true
|
||||
set_strikes "daemon_restarted_flag" "true" "$WATCHDOG_STATE_FILE"
|
||||
if /etc/rc.d/rc.docker restart >/dev/null 2>&1; then
|
||||
log "Docker daemon restart issued — waiting ${WATCHDOG_DAEMON_RESTART_WAIT}s..."
|
||||
sleep "$WATCHDOG_DAEMON_RESTART_WAIT"
|
||||
|
||||
if timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
|
||||
success "Docker daemon restarted successfully ✅"
|
||||
notify "Docker daemon restarted successfully on $(hostname)" "Docker Watchdog" "normal"
|
||||
WATCHDOG_DAEMON_STRIKES=0
|
||||
WATCHDOG_DAEMON_RESTARTED=false
|
||||
set_strikes "daemon_strikes" 0 "$WATCHDOG_STATE_FILE"
|
||||
set_strikes "daemon_restarted_flag" "false" "$WATCHDOG_STATE_FILE"
|
||||
return 0
|
||||
else
|
||||
error "Docker daemon did not recover after restart"
|
||||
queue_notify "Docker daemon restart failed on $(hostname) — system_watchdog.sh escalating" "critical"
|
||||
flush_notify
|
||||
return 1
|
||||
fi
|
||||
else
|
||||
error "Failed to issue Docker daemon restart — /etc/rc.d/rc.docker not found or failed"
|
||||
queue_notify "Docker daemon restart command failed on $(hostname) — manual intervention needed" "critical"
|
||||
flush_notify
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Single-Pass Monitoring Run ━━━
|
||||
# ==============================================================================================
|
||||
log "$ICON_WATCHDOG Docker watchdog — $MY_ID — $(date '+%H:%M:%S')"
|
||||
|
||||
CYCLE_START=$(date +%s)
|
||||
|
||||
# ── Per-run state ─────────────────────────────────────────────────────────────────────────
|
||||
NOTIFY_EVENTS=()
|
||||
T1_RESTARTS=0
|
||||
T1_WARNINGS=0
|
||||
T2_RESTARTS=0
|
||||
T2_WARNINGS=0
|
||||
|
||||
# OOM handled set — tracks containers already handled for OOM this cycle
|
||||
# Prevents restart loop from OOMKilled flag persisting after restart
|
||||
declare -A OOM_HANDLED
|
||||
|
||||
# Rebuild ignore map each cycle (config may have changed)
|
||||
declare -A IGNORE_MAP
|
||||
for c in "${WATCHDOG_SCAN_IGNORE[@]:-}"; do
|
||||
[[ -n "$c" ]] && IGNORE_MAP["$c"]=1
|
||||
done
|
||||
|
||||
# ── Docker daemon health check — first check every run ──────────────────────────────────
|
||||
# If daemon is hung all container operations will fail — check first, skip run if down
|
||||
if ! check_docker_daemon; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Parity check — skip restarts during parity ───────────────────────────────────────────
|
||||
if is_parity_running; then
|
||||
log "Parity check in progress — skipping restart actions this run"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── RAM emergency check — resource_watchdog.sh managing containers ────────────────────────
|
||||
# If resource_watchdog.sh has triggered a hard RAM shutdown, defer all container
|
||||
# management this run to prevent undoing the emergency stop and re-pressuring RAM.
|
||||
if ! check_system_watchdog_state; then
|
||||
MEM_KB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
|
||||
MEM_GB=$(( MEM_KB / 1024 / 1024 ))
|
||||
warn "RAM emergency active (${MEM_GB}GB free) — resource_watchdog.sh managing containers"
|
||||
warn "Deferring all container restart logic this run"
|
||||
log "Waiting for RAM to recover above ${RW_RAM_RECOVER_GB:-20}GB before resuming"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Startup grace period ──────────────────────────────────────────────────────────────────
|
||||
UPTIME_SECONDS=$(awk '{print int($1)}' /proc/uptime)
|
||||
IN_GRACE_PERIOD=false
|
||||
[[ "$UPTIME_SECONDS" -lt "$WATCHDOG_STARTUP_GRACE" ]] && IN_GRACE_PERIOD=true
|
||||
|
||||
# ==========================================================================================
|
||||
# ── TIER 1 — Strict Monitoring ────────────────────────────────────────────────────────────
|
||||
# ==========================================================================================
|
||||
|
||||
# ── Required containers ───────────────────────────────────────────────────────────────────
|
||||
# Must always be running — strike system before restart, skip list after limit
|
||||
if [[ ${#WATCHDOG_REQUIRED_CONTAINERS[@]} -gt 0 ]]; then
|
||||
for container in "${WATCHDOG_REQUIRED_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' \
|
||||
"$container" 2>/dev/null)
|
||||
|
||||
if is_skipped "$container"; then
|
||||
if [[ "$STATUS" == "true" ]]; then
|
||||
# Container recovered — remove from skip list and clear its history
|
||||
remove_from_skip_list "$container"
|
||||
set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
||||
sed -i "/^${container}|/d" "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null
|
||||
queue_notify "$container recovered on $(hostname) — removed from skip list" "normal"
|
||||
else
|
||||
warn "$container — on skip list, manual intervention needed"
|
||||
fi
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$STATUS" == "true" ]]; then
|
||||
# Running — clear any strikes
|
||||
set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
||||
else
|
||||
STRIKES=$(get_strikes "$container" "$WATCHDOG_STATE_FILE")
|
||||
STRIKES=$(( STRIKES + 1 ))
|
||||
set_strikes "$container" "$STRIKES" "$WATCHDOG_STATE_FILE"
|
||||
warn "$container — not running (strike $STRIKES/$SYS_WATCHDOG_STRIKE_LIMIT)"
|
||||
((T1_WARNINGS++))
|
||||
|
||||
if [[ "$STRIKES" -ge "$SYS_WATCHDOG_STRIKE_LIMIT" ]]; then
|
||||
result=0
|
||||
safe_restart "$container" "required container down" || result=$?
|
||||
case $result in
|
||||
0) set_strikes "$container" 0 "$WATCHDOG_STATE_FILE"
|
||||
((T1_RESTARTS++))
|
||||
queue_notify "$container was down and restarted on $(hostname)" "warning" ;;
|
||||
2) : ;; # Added to skip list — already notified
|
||||
*) queue_notify "$container failed to restart on $(hostname)" "warning" ;;
|
||||
esac
|
||||
fi
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Memory and CPU monitoring ─────────────────────────────────────────────────────────────
|
||||
if [[ ${#WATCHDOG_CONTAINERS[@]} -gt 0 ]]; then
|
||||
STATS=$(timeout "$DOCKER_TIMEOUT" docker stats --no-stream \
|
||||
--format "{{.Name}}|{{.MemUsage}}|{{.CPUPerc}}" 2>/dev/null)
|
||||
TOTAL_CORES=$(nproc 2>/dev/null || echo 1)
|
||||
|
||||
for container in "${!WATCHDOG_CONTAINERS[@]}"; do
|
||||
MEM_LIMIT_MB="${WATCHDOG_CONTAINERS[$container]}"
|
||||
CONTAINER_STATS=$(echo "$STATS" | grep "^${container}|" | head -1)
|
||||
[[ -z "$CONTAINER_STATS" ]] && continue
|
||||
|
||||
# ── Memory ────────────────────────────────────────────────────────────────────────
|
||||
MEM_USAGE=$(echo "$CONTAINER_STATS" | cut -d'|' -f2 | awk '{print $1}')
|
||||
MEM_UNIT=$(echo "$MEM_USAGE" | grep -oE '[A-Za-z]+')
|
||||
MEM_VALUE=$(echo "$MEM_USAGE" | grep -oE '[0-9.]+')
|
||||
|
||||
case "$MEM_UNIT" in
|
||||
GiB|GB) MEM_MB=$(awk "BEGIN {printf \"%.0f\", $MEM_VALUE * 1024}") ;;
|
||||
MiB|MB) MEM_MB=$(awk "BEGIN {printf \"%.0f\", $MEM_VALUE}") ;;
|
||||
KiB|KB) MEM_MB=$(awk "BEGIN {printf \"%.0f\", $MEM_VALUE / 1024}") ;;
|
||||
*) MEM_MB=0 ;;
|
||||
esac
|
||||
|
||||
# Soft memory threshold — warn when approaching hard limit
|
||||
SOFT_MEM_MB=$(awk "BEGIN {printf \"%.0f\", $MEM_LIMIT_MB * $SOFT_MEM_THRESHOLD / 100}")
|
||||
if [[ "$MEM_MB" -ge "$MEM_LIMIT_MB" ]]; then
|
||||
# Hard limit exceeded — immediate restart
|
||||
error "$container — memory ${MEM_MB}MB exceeded hard limit ${MEM_LIMIT_MB}MB"
|
||||
safe_restart "$container" "memory hard limit exceeded"
|
||||
((T1_RESTARTS++))
|
||||
queue_notify "$container exceeded memory hard limit on $(hostname) — restarted" "warning"
|
||||
elif [[ "$MEM_MB" -ge "$SOFT_MEM_MB" ]]; then
|
||||
# Soft threshold — warn only, no restart
|
||||
warn "$container — memory ${MEM_MB}MB approaching limit (${SOFT_MEM_THRESHOLD}% of ${MEM_LIMIT_MB}MB)"
|
||||
((T1_WARNINGS++))
|
||||
fi
|
||||
|
||||
# ── CPU ───────────────────────────────────────────────────────────────────────────
|
||||
CPU_RAW=$(echo "$CONTAINER_STATS" | cut -d'|' -f3 | tr -d '%')
|
||||
CPU_NORM=$(awk "BEGIN {printf \"%.1f\", $CPU_RAW / $TOTAL_CORES}")
|
||||
CPU_INT=$(printf "%.0f" "$CPU_NORM")
|
||||
|
||||
if [[ "$CPU_INT" -ge "$HARD_CPU_THRESHOLD" ]]; then
|
||||
# Hard CPU threshold — strike system → restart
|
||||
CPU_STRIKES=$(get_strikes "${container}_cpu" "$WATCHDOG_STATE_FILE")
|
||||
CPU_STRIKES=$(( CPU_STRIKES + 1 ))
|
||||
set_strikes "${container}_cpu" "$CPU_STRIKES" "$WATCHDOG_STATE_FILE"
|
||||
warn "$container — CPU ${CPU_NORM}% (strike $CPU_STRIKES/$CPU_FAIL_LIMIT)"
|
||||
if [[ "$CPU_STRIKES" -ge "$CPU_FAIL_LIMIT" ]]; then
|
||||
safe_restart "$container" "CPU hard threshold exceeded ${CPU_NORM}%"
|
||||
set_strikes "${container}_cpu" 0 "$WATCHDOG_STATE_FILE"
|
||||
((T1_RESTARTS++))
|
||||
queue_notify "$container CPU ${CPU_NORM}% on $(hostname) — restarted" "warning"
|
||||
fi
|
||||
elif [[ "$CPU_INT" -ge "$SOFT_CPU_THRESHOLD" ]]; then
|
||||
# Soft CPU threshold — warn only, no restart, clear strikes
|
||||
warn "$container — CPU ${CPU_NORM}% (above soft threshold ${SOFT_CPU_THRESHOLD}%)"
|
||||
((T1_WARNINGS++))
|
||||
set_strikes "${container}_cpu" 0 "$WATCHDOG_STATE_FILE"
|
||||
else
|
||||
# Normal — clear CPU strikes
|
||||
set_strikes "${container}_cpu" 0 "$WATCHDOG_STATE_FILE"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── HTTP responsiveness ───────────────────────────────────────────────────────────────────
|
||||
if [[ ${#WATCHDOG_CONTAINER_URLS[@]} -gt 0 ]]; then
|
||||
for container in "${!WATCHDOG_CONTAINER_URLS[@]}"; do
|
||||
URL="${WATCHDOG_CONTAINER_URLS[$container]}"
|
||||
if curl -sf --max-time "$CURL_TIMEOUT" "$URL" >/dev/null 2>&1; then
|
||||
set_strikes "${container}_http" 0 "$WATCHDOG_STATE_FILE"
|
||||
else
|
||||
HTTP_STRIKES=$(get_strikes "${container}_http" "$WATCHDOG_STATE_FILE")
|
||||
HTTP_STRIKES=$(( HTTP_STRIKES + 1 ))
|
||||
set_strikes "${container}_http" "$HTTP_STRIKES" "$WATCHDOG_STATE_FILE"
|
||||
warn "$container — not responding at $URL (strike $HTTP_STRIKES/$RESP_FAIL_LIMIT)"
|
||||
((T1_WARNINGS++))
|
||||
if [[ "$HTTP_STRIKES" -ge "$RESP_FAIL_LIMIT" ]]; then
|
||||
result=0
|
||||
safe_restart "$container" "HTTP unresponsive at $URL" || result=$?
|
||||
if [[ $result -eq 0 ]]; then
|
||||
set_strikes "${container}_http" 0 "$WATCHDOG_STATE_FILE"
|
||||
((T1_RESTARTS++))
|
||||
queue_notify "$container unresponsive at $URL on $(hostname) — restarted" "warning"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ==========================================================================================
|
||||
# ── TIER 2 — Global Health Scan ───────────────────────────────────────────────────────────
|
||||
# ==========================================================================================
|
||||
if [[ "$WATCHDOG_SCAN_ALL" == "true" ]]; then
|
||||
|
||||
ALL_CONTAINERS=$(timeout "$DOCKER_TIMEOUT" docker ps --format "{{.Names}}" 2>/dev/null)
|
||||
|
||||
# ── Unhealthy containers ──────────────────────────────────────────────────────────────
|
||||
if [[ "$WATCHDOG_RESTART_UNHEALTHY" == "true" ]]; then
|
||||
UNHEALTHY=$(timeout "$DOCKER_TIMEOUT" docker ps \
|
||||
--filter health=unhealthy --format "{{.Names}}" 2>/dev/null)
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
[[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
|
||||
is_skipped "$container" && continue
|
||||
error "$container — Docker HEALTHCHECK unhealthy"
|
||||
((T2_WARNINGS++))
|
||||
result=0
|
||||
safe_restart "$container" "unhealthy health status" || result=$?
|
||||
if [[ $result -eq 0 ]]; then
|
||||
((T2_RESTARTS++))
|
||||
queue_notify "$container unhealthy on $(hostname) — restarted" "warning"
|
||||
fi
|
||||
done <<< "$UNHEALTHY"
|
||||
fi
|
||||
|
||||
# ── OOM killed ────────────────────────────────────────────────────────────────────────
|
||||
# OOMKilled flag persists after restart — track handled containers per-cycle
|
||||
# to prevent the same container triggering a restart loop every cycle
|
||||
if [[ "$WATCHDOG_NOTIFY_OOM" == "true" ]]; then
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
[[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
|
||||
is_skipped "$container" && continue
|
||||
[[ -n "${OOM_HANDLED[$container]:-}" ]] && continue # already handled this cycle
|
||||
OOM=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
|
||||
'{{.State.OOMKilled}}' "$container" 2>/dev/null)
|
||||
if [[ "$OOM" == "true" ]]; then
|
||||
error "$container — OOM killed by kernel"
|
||||
((T2_WARNINGS++))
|
||||
OOM_HANDLED["$container"]=1
|
||||
result=0
|
||||
safe_restart "$container" "OOM killed" || result=$?
|
||||
if [[ $result -eq 0 ]]; then
|
||||
((T2_RESTARTS++))
|
||||
queue_notify "$container OOM killed on $(hostname) — restarted" "warning"
|
||||
fi
|
||||
fi
|
||||
done <<< "$ALL_CONTAINERS"
|
||||
fi
|
||||
|
||||
# ── Crash loop detection ──────────────────────────────────────────────────────────────
|
||||
# Tracks Docker's own RestartCount climbing between cycles.
|
||||
# Below WATCHDOG_CRASH_LIMIT: notify only — Docker's restart policy is handling it.
|
||||
# At or above WATCHDOG_CRASH_LIMIT: safe_restart() which will add to skip list
|
||||
# if WATCHDOG_CONTAINER_RESTART_LIMIT is also hit — ensures eventual quarantine.
|
||||
if [[ "$WATCHDOG_NOTIFY_CRASHLOOP" == "true" ]]; then
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
[[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
|
||||
is_skipped "$container" && continue
|
||||
RESTART_COUNT=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
|
||||
'{{.RestartCount}}' "$container" 2>/dev/null || echo 0)
|
||||
PREV_COUNT=$(grep "^${container}_docker:" "$WATCHDOG_STATE_FILE" \
|
||||
2>/dev/null | cut -d: -f2 || echo 0)
|
||||
set_strikes "${container}_docker" "$RESTART_COUNT" "$WATCHDOG_STATE_FILE"
|
||||
if [[ "$RESTART_COUNT" -gt "$PREV_COUNT" && "$RESTART_COUNT" -gt 0 ]]; then
|
||||
((T2_WARNINGS++))
|
||||
if [[ "$RESTART_COUNT" -ge "$WATCHDOG_CRASH_LIMIT" ]]; then
|
||||
error "$container — crash loop CRITICAL: $RESTART_COUNT restarts"
|
||||
# Attempt restart via safe_restart — will add to skip list if over limit
|
||||
result=0
|
||||
safe_restart "$container" "crash loop — $RESTART_COUNT restarts" || result=$?
|
||||
case $result in
|
||||
0) ((T2_RESTARTS++))
|
||||
queue_notify "$container crash loop on $(hostname) — restarted" "critical" ;;
|
||||
2) : ;; # Added to skip list
|
||||
*) queue_notify "$container crash loop CRITICAL on $(hostname) — manual intervention needed" "critical" ;;
|
||||
esac
|
||||
else
|
||||
warn "$container — restarted since last check (Docker count: $RESTART_COUNT)"
|
||||
queue_notify "$container restarted on $(hostname) — Docker count: $RESTART_COUNT" "warning"
|
||||
fi
|
||||
fi
|
||||
done <<< "$ALL_CONTAINERS"
|
||||
fi
|
||||
|
||||
# ── Dead containers ───────────────────────────────────────────────────────────────────
|
||||
# Routes through safe_restart() — ensures restart loop protection applies
|
||||
if [[ "$WATCHDOG_RESTART_DEAD" == "true" ]]; then
|
||||
DEAD=$(timeout "$DOCKER_TIMEOUT" docker ps -a \
|
||||
--filter status=dead --format "{{.Names}}" 2>/dev/null)
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
[[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
|
||||
is_skipped "$container" && continue
|
||||
error "$container — dead"
|
||||
((T2_WARNINGS++))
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
timeout "$DOCKER_TIMEOUT" docker rm "$container" >/dev/null 2>&1
|
||||
fi
|
||||
result=0
|
||||
safe_restart "$container" "dead container" || result=$?
|
||||
if [[ $result -eq 0 ]]; then
|
||||
((T2_RESTARTS++))
|
||||
queue_notify "$container was dead on $(hostname) — removed and restarted" "warning"
|
||||
fi
|
||||
done <<< "$DEAD"
|
||||
fi
|
||||
|
||||
# ── Unexpected exits ──────────────────────────────────────────────────────────────────
|
||||
# Only non-zero exit codes — exit 0 is a clean stop, not a crash
|
||||
# Skips containers already covered by WATCHDOG_REQUIRED_CONTAINERS (handled in Tier 1)
|
||||
if [[ "$WATCHDOG_RESTART_CRASHED" == "true" ]]; then
|
||||
CRASHED=$(timeout "$DOCKER_TIMEOUT" docker ps -a \
|
||||
--filter status=exited \
|
||||
--format "{{.Names}}|{{.Status}}" 2>/dev/null | \
|
||||
grep -v "Exited (0)")
|
||||
while IFS='|' read -r container status; do
|
||||
[[ -z "$container" ]] && continue
|
||||
[[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
|
||||
is_skipped "$container" && continue
|
||||
# Skip containers already monitored by required containers (Tier 1)
|
||||
already_required=false
|
||||
for req in "${WATCHDOG_REQUIRED_CONTAINERS[@]:-}"; do
|
||||
[[ "$container" == "$req" ]] && already_required=true && break
|
||||
done
|
||||
[[ "$already_required" == true ]] && continue
|
||||
error "$container — $status (unexpected exit)"
|
||||
((T2_WARNINGS++))
|
||||
result=0
|
||||
safe_restart "$container" "unexpected exit: $status" || result=$?
|
||||
if [[ $result -eq 0 ]]; then
|
||||
((T2_RESTARTS++))
|
||||
queue_notify "$container crashed on $(hostname) ($status) — restarted" "warning"
|
||||
fi
|
||||
done <<< "$CRASHED"
|
||||
fi
|
||||
|
||||
fi # WATCHDOG_SCAN_ALL
|
||||
|
||||
|
||||
# ── Send notifications ────────────────────────────────────────────────────────────────────
|
||||
flush_notify
|
||||
|
||||
# ── Cycle summary — quiet when healthy ───────────────────────────────────────────────────
|
||||
TOTAL_RESTARTS=$(( T1_RESTARTS + T2_RESTARTS ))
|
||||
TOTAL_WARNINGS=$(( T1_WARNINGS + T2_WARNINGS ))
|
||||
CYCLE_END=$(date +%s)
|
||||
|
||||
if [[ "$TOTAL_RESTARTS" -gt 0 || "$TOTAL_WARNINGS" -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_WATCHDOG Docker Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_WATCHDOG T1: $T1_RESTARTS restarts / $T1_WARNINGS warnings"
|
||||
echo "$ICON_WATCHDOG T2: $T2_RESTARTS restarts / $T2_WARNINGS warnings"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( CYCLE_END - CYCLE_START )))"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
log "All healthy ✅ ($(date '+%H:%M:%S'))"
|
||||
fi
|
||||
Executable
+585
@@ -0,0 +1,585 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Resource Manager ===========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pressure reduction layer — detects rising system load and reduces it before
|
||||
# things break. Called by watchdog_orchestrator.sh every minute as a single-
|
||||
# pass run. The middle layer between docker_watchdog.sh (fixes broken
|
||||
# containers) and system_watchdog.sh (reboots). Does neither of those things.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three-Level Pressure Response
|
||||
#
|
||||
# Level 1 — SOFT (RAM < RW_RAM_SOFT_GB OR load > RW_LOAD_SOFT_MULTIPLIER × cores):
|
||||
# Throttle SABnzbd download speed to RW_SABNZBD_SPEED_SOFT.
|
||||
# Throttle qBittorrent download to RW_QBIT_DL_SOFT KB/s.
|
||||
#
|
||||
# Level 2 — MEDIUM (RAM < RW_RAM_MEDIUM_GB OR load > RW_LOAD_MEDIUM_MULTIPLIER × cores):
|
||||
# Further throttle SABnzbd + qBittorrent to medium limits.
|
||||
# docker pause RW_PAUSE_CONTAINERS — suspend without losing state, instantly reversible.
|
||||
#
|
||||
# Level 3 — HARD (RAM < RW_RAM_HARD_GB):
|
||||
# docker stop RW_STOP_CONTAINERS — optional/heavy services (games, LocalAI, etc.).
|
||||
# Write mem_shutdown_active=true → signals docker_watchdog to defer container restarts.
|
||||
#
|
||||
# Recovery
|
||||
# Pressure must stay below current threshold for RW_RECOVER_CYCLES consecutive
|
||||
# runs before restoring. De-escalates one level at a time — prevents re-triggering
|
||||
# immediately after recovery. Level 3 additionally requires RAM >= RW_RAM_RECOVER_GB
|
||||
# before containers are un-stopped.
|
||||
#
|
||||
# Coordination with docker_watchdog.sh
|
||||
# At level 3, writes mem_shutdown_active=true to RW_STATE_FILE.
|
||||
# docker_watchdog.sh reads this and defers all container restart logic.
|
||||
# Without this, docker_watchdog would immediately restart containers that were
|
||||
# just stopped to free RAM — defeating the purpose of level 3.
|
||||
# Cleared when pressure resolves and containers are restarted.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# docker pause/stop require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs from racing on state file writes.
|
||||
#
|
||||
# RW_CRITICAL_CONTAINERS
|
||||
# Containers listed here are never paused or stopped regardless of pressure level.
|
||||
#
|
||||
# RW_ENABLED Flag
|
||||
# Set RW_ENABLED=false to disable the entire script without removing it from
|
||||
# the orchestrator schedule.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
# RW_ENABLED, RW_STATE_FILE
|
||||
# RW_RAM_SOFT_GB, RW_RAM_MEDIUM_GB, RW_RAM_HARD_GB, RW_RAM_RECOVER_GB
|
||||
# RW_LOAD_SOFT_MULTIPLIER, RW_LOAD_MEDIUM_MULTIPLIER
|
||||
# RW_RECOVER_CYCLES
|
||||
# RW_SABNZBD_ENABLED, RW_SABNZBD_SPEED_SOFT, RW_SABNZBD_SPEED_MEDIUM
|
||||
# RW_QBIT_ENABLED, RW_QBIT_DL_SOFT, RW_QBIT_DL_MEDIUM
|
||||
# RW_CRITICAL_CONTAINERS — never paused or stopped regardless of pressure
|
||||
#
|
||||
# host*.conf (aliased by detect_hosts())
|
||||
# HOST*_RW_PAUSE_CONTAINERS — docker pause at medium pressure
|
||||
# HOST*_RW_STOP_CONTAINERS — docker stop at hard pressure
|
||||
# HOST*_SABNZBD_URL, HOST*_SABNZBD_API_KEY
|
||||
# HOST*_QBIT_URL, HOST*_QBIT_USERNAME, HOST*_QBIT_PASSWORD
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# RW_STATE_FILE
|
||||
# Pressure level, recovery cycle count, stopped container list, and the
|
||||
# mem_shutdown_active coordination flag read by docker_watchdog.sh.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# resource_watchdog.sh
|
||||
# Single-pass pressure check. Apply actions if threshold crossed. Silent if below.
|
||||
#
|
||||
# resource_watchdog.sh --dry-run
|
||||
# Show current pressure level and what would be throttled/paused/stopped. No changes.
|
||||
#
|
||||
# resource_watchdog.sh --status
|
||||
# Show current pressure level, active actions, recovery cycle count, stopped containers.
|
||||
#
|
||||
# resource_watchdog.sh --log
|
||||
# Verbose per-check output — show RAM, load, each threshold comparison, each action.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${RW_ENABLED:-true}" != "true" ]]; then
|
||||
echo "Resource Manager disabled (RW_ENABLED=false)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
DOCKER_TIMEOUT=15
|
||||
|
||||
touch "$RW_STATE_FILE" 2>/dev/null || {
|
||||
error "Cannot create state file: $RW_STATE_FILE"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ── Exit Trap — restart containers stopped this run if script crashes ──────────────────────────
|
||||
declare -a _RW_TRAP_STOPPED=()
|
||||
|
||||
_rw_trap_restart_stopped() {
|
||||
[[ ${#_RW_TRAP_STOPPED[@]} -eq 0 ]] && return
|
||||
for c in "${_RW_TRAP_STOPPED[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
if docker inspect "$c" >/dev/null 2>&1; then
|
||||
warn "Exit trap: restarting $c (stopped but state not persisted)"
|
||||
docker start "$c" >/dev/null 2>&1 || warn " Failed to restart $c"
|
||||
fi
|
||||
done
|
||||
}
|
||||
trap _rw_trap_restart_stopped EXIT
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ State Helpers ━━━
|
||||
# ==============================================================================================
|
||||
# rm_state_get/set use : separator for RM-internal state
|
||||
# rm_state_get_eq/set_eq use = separator for docker_watchdog coordination flags
|
||||
|
||||
rm_state_get() {
|
||||
grep -E "^${1}:" "$RW_STATE_FILE" 2>/dev/null | cut -d: -f2-
|
||||
}
|
||||
|
||||
rm_state_set() {
|
||||
local key="$1" val="$2"
|
||||
grep -vE "^${key}:" "$RW_STATE_FILE" 2>/dev/null > "${RW_STATE_FILE}.tmp"
|
||||
echo "${key}:${val}" >> "${RW_STATE_FILE}.tmp"
|
||||
mv "${RW_STATE_FILE}.tmp" "$RW_STATE_FILE"
|
||||
}
|
||||
|
||||
rm_state_get_eq() {
|
||||
grep -E "^${1}=" "$RW_STATE_FILE" 2>/dev/null | cut -d= -f2-
|
||||
}
|
||||
|
||||
rm_state_set_eq() {
|
||||
local key="$1" val="$2"
|
||||
grep -vE "^${key}=" "$RW_STATE_FILE" 2>/dev/null > "${RW_STATE_FILE}.tmp"
|
||||
echo "${key}=${val}" >> "${RW_STATE_FILE}.tmp"
|
||||
mv "${RW_STATE_FILE}.tmp" "$RW_STATE_FILE"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Load State ━━━
|
||||
# ==============================================================================================
|
||||
CURRENT_LEVEL=$(rm_state_get "rm_action_level"); CURRENT_LEVEL=${CURRENT_LEVEL:-0}
|
||||
RECOVER_CYCLES=$(rm_state_get "rm_recover_cycles"); RECOVER_CYCLES=${RECOVER_CYCLES:-0}
|
||||
PAUSED_LIST=$(rm_state_get "rm_paused_containers"); PAUSED_LIST=${PAUSED_LIST:-""}
|
||||
STOPPED_LIST=$(rm_state_get "rm_stopped_containers"); STOPPED_LIST=${STOPPED_LIST:-""}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pressure Calculation ━━━
|
||||
# ==============================================================================================
|
||||
TOTAL_CORES=$(nproc)
|
||||
MEM_KB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
|
||||
MEM_GB=$(( MEM_KB / 1024 / 1024 ))
|
||||
LOAD=$(awk '{print $1}' /proc/loadavg)
|
||||
LOAD_INT=$(printf "%.0f" "$LOAD")
|
||||
RW_LOAD_SOFT_THRESH=$(awk "BEGIN{printf \"%.0f\", $TOTAL_CORES * ${RW_LOAD_SOFT_MULTIPLIER:-2.0}}")
|
||||
RW_LOAD_MEDIUM_THRESH=$(awk "BEGIN{printf \"%.0f\", $TOTAL_CORES * ${RW_LOAD_MEDIUM_MULTIPLIER:-3.0}}")
|
||||
|
||||
TARGET_LEVEL=0
|
||||
TARGET_REASON=""
|
||||
if [[ "$MEM_GB" -lt "${RW_RAM_HARD_GB:-6}" ]]; then
|
||||
TARGET_LEVEL=3
|
||||
TARGET_REASON="RAM ${MEM_GB}GB < hard threshold ${RW_RAM_HARD_GB}GB"
|
||||
elif [[ "$MEM_GB" -lt "${RW_RAM_MEDIUM_GB:-8}" ]] || [[ "$LOAD_INT" -ge "$RW_LOAD_MEDIUM_THRESH" ]]; then
|
||||
TARGET_LEVEL=2
|
||||
[[ "$MEM_GB" -lt "${RW_RAM_MEDIUM_GB:-8}" ]] && TARGET_REASON="RAM ${MEM_GB}GB < medium threshold ${RW_RAM_MEDIUM_GB}GB"
|
||||
[[ "$LOAD_INT" -ge "$RW_LOAD_MEDIUM_THRESH" ]] && TARGET_REASON="${TARGET_REASON:+$TARGET_REASON, }load ${LOAD} >= medium threshold ${RW_LOAD_MEDIUM_THRESH}"
|
||||
elif [[ "$MEM_GB" -lt "${RW_RAM_SOFT_GB:-12}" ]] || [[ "$LOAD_INT" -ge "$RW_LOAD_SOFT_THRESH" ]]; then
|
||||
TARGET_LEVEL=1
|
||||
[[ "$MEM_GB" -lt "${RW_RAM_SOFT_GB:-12}" ]] && TARGET_REASON="RAM ${MEM_GB}GB < soft threshold ${RW_RAM_SOFT_GB}GB"
|
||||
[[ "$LOAD_INT" -ge "$RW_LOAD_SOFT_THRESH" ]] && TARGET_REASON="${TARGET_REASON:+$TARGET_REASON, }load ${LOAD} >= soft threshold ${RW_LOAD_SOFT_THRESH}"
|
||||
fi
|
||||
|
||||
LEVEL_NAMES=("normal" "soft" "medium" "hard")
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY RESOURCE MANAGER STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
echo "── Current State ──"
|
||||
echo " Action level: $CURRENT_LEVEL (${LEVEL_NAMES[$CURRENT_LEVEL]:-unknown})"
|
||||
echo " Target level: $TARGET_LEVEL (${LEVEL_NAMES[$TARGET_LEVEL]:-unknown})"
|
||||
echo " Recover cycles: $RECOVER_CYCLES / ${RW_RECOVER_CYCLES:-3}"
|
||||
[[ -n "$PAUSED_LIST" ]] && echo " Paused: $PAUSED_LIST"
|
||||
[[ -n "$STOPPED_LIST" ]] && echo " Stopped: $STOPPED_LIST"
|
||||
MEM_SHUTDOWN_ACTIVE=$(rm_state_get_eq "mem_shutdown_active")
|
||||
[[ "$MEM_SHUTDOWN_ACTIVE" == "true" ]] && warn " docker_watchdog DEFERRED (mem_shutdown_active=true)"
|
||||
echo ""
|
||||
echo "── System Pressure ──"
|
||||
echo " RAM free: ${MEM_GB}GB (soft:<${RW_RAM_SOFT_GB} medium:<${RW_RAM_MEDIUM_GB} hard:<${RW_RAM_HARD_GB} recover:>=${RW_RAM_RECOVER_GB})"
|
||||
echo " Load avg: ${LOAD} (soft:>=${RW_LOAD_SOFT_THRESH} medium:>=${RW_LOAD_MEDIUM_THRESH} cores:${TOTAL_CORES})"
|
||||
echo ""
|
||||
echo "── Configuration ──"
|
||||
echo " SABnzbd throttle: ${RW_SABNZBD_ENABLED:-true} soft=${RW_SABNZBD_SPEED_SOFT} medium=${RW_SABNZBD_SPEED_MEDIUM}"
|
||||
echo " qBit throttle: ${RW_QBIT_ENABLED:-true} soft=${RW_QBIT_DL_SOFT}KB/s medium=${RW_QBIT_DL_MEDIUM}KB/s"
|
||||
echo ""
|
||||
echo "── Container Lists (this host) ──"
|
||||
echo " Pause at medium: ${RW_PAUSE_CONTAINERS[*]:-none configured}"
|
||||
echo " Stop at hard: ${RW_STOP_CONTAINERS[*]:-none configured}"
|
||||
echo " Critical (never touched): ${RW_CRITICAL_CONTAINERS[*]:-none}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Critical Container Guard ━━━
|
||||
# ==============================================================================================
|
||||
# Returns 0 if container is safe to pause/stop, 1 if it is critical
|
||||
is_critical() {
|
||||
local container="$1"
|
||||
for c in "${RW_CRITICAL_CONTAINERS[@]:-}"; do
|
||||
[[ "$c" == "$container" ]] && return 1
|
||||
done
|
||||
return 0
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ SABnzbd API ━━━
|
||||
# ==============================================================================================
|
||||
sabnzbd_set_speed() {
|
||||
local speed="$1"
|
||||
[[ "${RW_SABNZBD_ENABLED:-true}" != "true" ]] && return 0
|
||||
[[ -z "$SABNZBD_URL" || -z "$SABNZBD_API_KEY" ]] && return 0
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would set SABnzbd speed to $speed"
|
||||
return 0
|
||||
fi
|
||||
|
||||
curl -sf --max-time 10 \
|
||||
"${SABNZBD_URL}/api?mode=config&name=speedlimit&value=${speed}&apikey=${SABNZBD_API_KEY}" \
|
||||
>/dev/null 2>&1 && log "SABnzbd speed → $speed" || warn "SABnzbd API call failed"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ qBittorrent API ━━━
|
||||
# ==============================================================================================
|
||||
QBIT_COOKIE="/tmp/rm_qbit_cookie.txt"
|
||||
|
||||
qbit_login() {
|
||||
[[ "${RW_QBIT_ENABLED:-true}" != "true" ]] && return 0
|
||||
[[ -z "$QBIT_URL" || -z "$QBIT_USERNAME" || -z "$QBIT_PASSWORD" ]] && return 0
|
||||
|
||||
curl -sf --max-time 10 -c "$QBIT_COOKIE" \
|
||||
-X POST "${QBIT_URL}/api/v2/auth/login" \
|
||||
-d "username=${QBIT_USERNAME}&password=${QBIT_PASSWORD}" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
qbit_set_dl_limit() {
|
||||
local kbps="$1" # KB/s — 0 = unlimited
|
||||
[[ "${RW_QBIT_ENABLED:-true}" != "true" ]] && return 0
|
||||
[[ -z "$QBIT_URL" ]] && return 0
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would set qBit download limit to ${kbps}KB/s"
|
||||
return 0
|
||||
fi
|
||||
|
||||
local bps=$(( kbps * 1024 ))
|
||||
qbit_login
|
||||
curl -sf --max-time 10 -b "$QBIT_COOKIE" \
|
||||
-X POST "${QBIT_URL}/api/v2/transfer/setDownloadLimit" \
|
||||
-d "limit=${bps}" >/dev/null 2>&1 && log "qBit download limit → ${kbps}KB/s" || warn "qBit API call failed"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Actions ━━━
|
||||
# ==============================================================================================
|
||||
|
||||
# Pause a list of containers — returns newline-separated list of actually-paused containers
|
||||
pause_containers() {
|
||||
local actually_paused=()
|
||||
for container in "$@"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
is_critical "$container" || { log "$container — critical, skipping pause"; continue; }
|
||||
|
||||
local status
|
||||
status=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Status}}' "$container" 2>/dev/null)
|
||||
if [[ "$status" != "running" ]]; then
|
||||
log "$container — not running (status: ${status:-unknown}), skipping pause"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would docker pause $container"
|
||||
actually_paused+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
if timeout "$DOCKER_TIMEOUT" docker pause "$container" >/dev/null 2>&1; then
|
||||
warn "Paused $container (medium pressure)"
|
||||
actually_paused+=("$container")
|
||||
else
|
||||
error "Failed to pause $container"
|
||||
fi
|
||||
done
|
||||
printf '%s,' "${actually_paused[@]}" | sed 's/,$//'
|
||||
}
|
||||
|
||||
# Unpause a comma-separated list of containers
|
||||
unpause_containers() {
|
||||
local IFS=','
|
||||
for container in $1; do
|
||||
[[ -z "$container" ]] && continue
|
||||
|
||||
local status
|
||||
status=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Status}}' "$container" 2>/dev/null)
|
||||
if [[ "$status" != "paused" ]]; then
|
||||
log "$container — not paused (status: ${status:-unknown}), skipping unpause"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would docker unpause $container"
|
||||
continue
|
||||
fi
|
||||
|
||||
if timeout "$DOCKER_TIMEOUT" docker unpause "$container" >/dev/null 2>&1; then
|
||||
warn "Unpaused $container (pressure reduced)"
|
||||
else
|
||||
error "Failed to unpause $container"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Stop a list of containers — returns comma-separated list of actually-stopped containers
|
||||
stop_containers() {
|
||||
local actually_stopped=()
|
||||
for container in "$@"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
is_critical "$container" || { log "$container — critical, skipping stop"; continue; }
|
||||
|
||||
local status
|
||||
status=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Status}}' "$container" 2>/dev/null)
|
||||
if [[ "$status" != "running" && "$status" != "paused" ]]; then
|
||||
log "$container — not running (status: ${status:-unknown}), skipping stop"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would docker stop $container"
|
||||
actually_stopped+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
if timeout "$DOCKER_TIMEOUT" docker stop "$container" >/dev/null 2>&1; then
|
||||
warn "Stopped $container (hard pressure)"
|
||||
actually_stopped+=("$container")
|
||||
_RW_TRAP_STOPPED+=("$container")
|
||||
else
|
||||
error "Failed to stop $container"
|
||||
fi
|
||||
done
|
||||
printf '%s,' "${actually_stopped[@]}" | sed 's/,$//'
|
||||
}
|
||||
|
||||
# Start a comma-separated list of containers (only those RM stopped)
|
||||
start_containers() {
|
||||
local IFS=','
|
||||
for container in $1; do
|
||||
[[ -z "$container" ]] && continue
|
||||
|
||||
local status
|
||||
status=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Status}}' "$container" 2>/dev/null)
|
||||
if [[ "$status" == "running" ]]; then
|
||||
log "$container — already running"
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would docker start $container"
|
||||
continue
|
||||
fi
|
||||
|
||||
if timeout "$DOCKER_TIMEOUT" docker start "$container" >/dev/null 2>&1; then
|
||||
warn "Started $container (pressure cleared)"
|
||||
else
|
||||
error "Failed to start $container"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Apply Level Actions ━━━
|
||||
# ==============================================================================================
|
||||
|
||||
apply_level_1() {
|
||||
log "Applying level 1 (soft) — throttling downloaders"
|
||||
sabnzbd_set_speed "${RW_SABNZBD_SPEED_SOFT:-50M}"
|
||||
qbit_set_dl_limit "${RW_QBIT_DL_SOFT:-51200}"
|
||||
}
|
||||
|
||||
apply_level_2() {
|
||||
log "Applying level 2 (medium) — throttling + pausing background containers"
|
||||
sabnzbd_set_speed "${RW_SABNZBD_SPEED_MEDIUM:-10M}"
|
||||
qbit_set_dl_limit "${RW_QBIT_DL_MEDIUM:-10240}"
|
||||
|
||||
if [[ ${#RW_PAUSE_CONTAINERS[@]} -gt 0 ]]; then
|
||||
local newly_paused
|
||||
newly_paused=$(pause_containers "${RW_PAUSE_CONTAINERS[@]}")
|
||||
# Merge with existing paused list (avoid duplicates on re-escalation)
|
||||
if [[ -n "$newly_paused" ]]; then
|
||||
if [[ -n "$PAUSED_LIST" ]]; then
|
||||
PAUSED_LIST="${PAUSED_LIST},${newly_paused}"
|
||||
else
|
||||
PAUSED_LIST="$newly_paused"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
apply_level_3() {
|
||||
log "Applying level 3 (hard) — stopping optional containers"
|
||||
sabnzbd_set_speed "${RW_SABNZBD_SPEED_MEDIUM:-10M}" # already at medium from level 2
|
||||
qbit_set_dl_limit "${RW_QBIT_DL_MEDIUM:-10240}"
|
||||
|
||||
if [[ ${#RW_STOP_CONTAINERS[@]} -gt 0 ]]; then
|
||||
local newly_stopped
|
||||
newly_stopped=$(stop_containers "${RW_STOP_CONTAINERS[@]}")
|
||||
if [[ -n "$newly_stopped" ]]; then
|
||||
if [[ -n "$STOPPED_LIST" ]]; then
|
||||
STOPPED_LIST="${STOPPED_LIST},${newly_stopped}"
|
||||
else
|
||||
STOPPED_LIST="$newly_stopped"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Signal docker_watchdog to defer container restarts
|
||||
rm_state_set_eq "mem_shutdown_active" "true"
|
||||
warn "mem_shutdown_active=true — docker_watchdog will defer restarts"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Restore Level Actions ━━━
|
||||
# ==============================================================================================
|
||||
|
||||
restore_level_3() {
|
||||
echo "Restoring from level 3 — starting stopped containers"
|
||||
if [[ -n "$STOPPED_LIST" ]]; then
|
||||
start_containers "$STOPPED_LIST"
|
||||
STOPPED_LIST=""
|
||||
fi
|
||||
rm_state_set_eq "mem_shutdown_active" "false"
|
||||
warn "mem_shutdown_active=false — docker_watchdog restoring normal operation"
|
||||
}
|
||||
|
||||
restore_level_2() {
|
||||
echo "Restoring from level 2 — unpausing containers"
|
||||
if [[ -n "$PAUSED_LIST" ]]; then
|
||||
unpause_containers "$PAUSED_LIST"
|
||||
PAUSED_LIST=""
|
||||
fi
|
||||
}
|
||||
|
||||
restore_level_1() {
|
||||
echo "Restoring from level 1 — removing downloader throttle"
|
||||
sabnzbd_set_speed "0"
|
||||
qbit_set_dl_limit 0
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pressure Decision ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Resource Manager — $MY_ID — $(date '+%H:%M:%S') ━━━"
|
||||
echo " RAM: ${MEM_GB}GB free Load: ${LOAD} Action: ${CURRENT_LEVEL} → ${TARGET_LEVEL} (${LEVEL_NAMES[$TARGET_LEVEL]:-unknown})"
|
||||
|
||||
if [[ "$TARGET_LEVEL" -gt "$CURRENT_LEVEL" ]]; then
|
||||
# ── Escalate ────────────────────────────────────────────────────────────────────────────
|
||||
warn "Pressure escalating to level $TARGET_LEVEL — $TARGET_REASON"
|
||||
notify "Resource Manager: pressure level $TARGET_LEVEL on $(hostname) ($MY_ID) — $TARGET_REASON" \
|
||||
"Resource Manager" "warning"
|
||||
|
||||
for (( lvl = CURRENT_LEVEL + 1; lvl <= TARGET_LEVEL; lvl++ )); do
|
||||
case "$lvl" in
|
||||
1) apply_level_1 ;;
|
||||
2) apply_level_2 ;;
|
||||
3) apply_level_3 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
rm_state_set "rm_action_level" "$TARGET_LEVEL"
|
||||
rm_state_set "rm_recover_cycles" 0
|
||||
|
||||
elif [[ "$TARGET_LEVEL" -lt "$CURRENT_LEVEL" ]]; then
|
||||
# ── Tracking recovery ───────────────────────────────────────────────────────────────────
|
||||
RECOVER_CYCLES=$(( RECOVER_CYCLES + 1 ))
|
||||
rm_state_set "rm_recover_cycles" "$RECOVER_CYCLES"
|
||||
log "Pressure at level $TARGET_LEVEL — recovery cycle $RECOVER_CYCLES/${RW_RECOVER_CYCLES:-3} before restoring level $CURRENT_LEVEL actions"
|
||||
|
||||
if [[ "$RECOVER_CYCLES" -ge "${RW_RECOVER_CYCLES:-3}" ]]; then
|
||||
# Level 3 de-escalation requires RAM above recover threshold
|
||||
if [[ "$CURRENT_LEVEL" -ge 3 && "$MEM_GB" -lt "${RW_RAM_RECOVER_GB:-20}" ]]; then
|
||||
warn "Level 3 restore blocked — RAM ${MEM_GB}GB still below recover threshold ${RW_RAM_RECOVER_GB}GB"
|
||||
else
|
||||
warn "Pressure sustained below level $CURRENT_LEVEL — restoring"
|
||||
case "$CURRENT_LEVEL" in
|
||||
3) restore_level_3 ;;
|
||||
2) restore_level_2 ;;
|
||||
1) restore_level_1 ;;
|
||||
esac
|
||||
|
||||
NEW_LEVEL=$(( CURRENT_LEVEL - 1 ))
|
||||
rm_state_set "rm_action_level" "$NEW_LEVEL"
|
||||
rm_state_set "rm_recover_cycles" 0
|
||||
rm_state_set "rm_paused_containers" "$PAUSED_LIST"
|
||||
rm_state_set "rm_stopped_containers" "$STOPPED_LIST"
|
||||
|
||||
if [[ "$NEW_LEVEL" -gt 0 ]]; then
|
||||
warn "De-escalated to level $NEW_LEVEL (${LEVEL_NAMES[$NEW_LEVEL]}) — ${RW_RECOVER_CYCLES:-3} more cycles to fully clear"
|
||||
else
|
||||
log "All pressure cleared — system at normal operation ✅"
|
||||
notify "Resource Manager: pressure resolved on $(hostname) ($MY_ID) — system back to normal" \
|
||||
"Resource Manager" "normal"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
else
|
||||
# ── Steady state ────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$CURRENT_LEVEL" -gt 0 ]]; then
|
||||
echo "Pressure holding at level $CURRENT_LEVEL — waiting for sustained recovery"
|
||||
else
|
||||
echo "System at normal pressure ✅"
|
||||
fi
|
||||
rm_state_set "rm_recover_cycles" 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Persist State ━━━
|
||||
# ==============================================================================================
|
||||
rm_state_set "rm_paused_containers" "$PAUSED_LIST"
|
||||
rm_state_set "rm_stopped_containers" "$STOPPED_LIST"
|
||||
trap - EXIT # state persisted — stopped containers recorded, trap no longer needed
|
||||
# Touch state file each run so docker_watchdog stale guard sees fresh mtime
|
||||
touch "$RW_STATE_FILE" 2>/dev/null
|
||||
Executable
+428
@@ -0,0 +1,428 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Storage Watchdog ===========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pool and storage health monitoring — catches runaway data growth before it
|
||||
# fills a pool. Runs as a single-pass script called by watchdog_orchestrator.sh
|
||||
# every cycle. Sits between docker_watchdog.sh (container health) and
|
||||
# system_watchdog.sh (last line of defense). Never reboots — detects, alerts,
|
||||
# and optionally remediates.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Appdata Size Monitoring
|
||||
# Two complementary checks run every cycle:
|
||||
#
|
||||
# Part 1 — Growth rate (zero-config catch-all):
|
||||
# Reads per-container dir totals via du, compares to previous cycle baseline.
|
||||
# Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused
|
||||
# *.log scan inside that container's dir. No per-container config required —
|
||||
# new containers are covered automatically. Baseline built on first cycle after
|
||||
# boot; growth detection active from cycle 2.
|
||||
#
|
||||
# Part 2 — Absolute log size:
|
||||
# Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB anywhere in
|
||||
# WATCHDOG_APPDATA_PATHS. Catches logs already large but no longer actively
|
||||
# growing. Independent strike counter per file.
|
||||
#
|
||||
# Strike System
|
||||
# Reuses the same strike pattern as CPU/HTTP checks in docker_watchdog.sh:
|
||||
#
|
||||
# Strike 1 — warn + notify: condition first detected this run
|
||||
# Strike 2 — warn + escalated notify: still present next cycle
|
||||
# Strike 3 (WATCHDOG_APPDATA_STRIKE_LIMIT) — action cycle:
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS=true → truncate *.log in-place, clear strikes
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS=false → critical notify, hold strikes until resolved
|
||||
# Condition resolves (growth stops / log drops below threshold) → strikes auto-clear
|
||||
#
|
||||
# Suppress Ceiling (WATCHDOG_APPDATA_SIZES)
|
||||
# Containers in HOST*_WATCHDOG_APPDATA_SIZES suppress growth warnings while
|
||||
# under their configured ceiling MB. Use ONLY when a container legitimately
|
||||
# holds large stable data and would otherwise false-alarm (e.g. Tdarr cache).
|
||||
# Zero-config growth detection covers everything else automatically.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Zero-config for new containers
|
||||
# Growth rate detection requires no per-container configuration. Add a game
|
||||
# server, spin up a new arr, install anything — it is monitored automatically
|
||||
# from the second cycle after it appears. The suppress ceiling in conf is the
|
||||
# exception, not the rule.
|
||||
#
|
||||
# Alert-only for data, truncate-only for logs
|
||||
# Non-log growth (databases, game saves, caches) is detected and alerted but
|
||||
# never touched. Only *.log / *.log.* files are candidates for truncation —
|
||||
# and only when WATCHDOG_APPDATA_TRUNCATE_LOGS=true. Truncation zeroes the
|
||||
# file in-place; the container keeps its open file handle, space is reclaimed
|
||||
# immediately. Never deletes.
|
||||
#
|
||||
# Strike before acting
|
||||
# One cycle of growth could be a legitimate library scan or game save burst.
|
||||
# Three consecutive cycles of growth is a runaway. The strike system separates
|
||||
# transient activity from sustained problems before any action fires.
|
||||
#
|
||||
# Silent when healthy
|
||||
# Produces no output when all checks pass. Loud only when something needs
|
||||
# attention.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WATCHDOG_CHECK_APPDATA
|
||||
# Master toggle for all appdata checks (default: true)
|
||||
#
|
||||
# WATCHDOG_APPDATA_PATHS
|
||||
# Array of paths to scan (e.g. "/mnt/docker-unraid/appdata")
|
||||
#
|
||||
# WATCHDOG_APPDATA_GROWTH_GB
|
||||
# Per-cycle growth threshold in GB — flag containers growing more than this (default: 2)
|
||||
#
|
||||
# WATCHDOG_APPDATA_LOG_MAX_GB
|
||||
# Absolute *.log file size alert threshold in GB (default: 2)
|
||||
#
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS
|
||||
# Auto-truncate oversized *.log files on action cycle (default: false)
|
||||
#
|
||||
# WATCHDOG_APPDATA_STRIKE_LIMIT
|
||||
# Consecutive cycles before action fires (default: 3)
|
||||
#
|
||||
# WATCHDOG_APPDATA_GROWTH_FILE
|
||||
# Per-container size baseline — /tmp resets on reboot (correct: stale baseline
|
||||
# after reboot would give false growth readings on first cycle)
|
||||
#
|
||||
# STORAGE_WATCHDOG_STATE_FILE
|
||||
# Strike counts for this script — /tmp resets on reboot
|
||||
#
|
||||
# host*.conf (aliased by detect_hosts())
|
||||
#
|
||||
# HOST*_WATCHDOG_APPDATA_SIZES
|
||||
# Per-container growth suppress ceilings in MB. Suppress growth alerts while
|
||||
# a container's dir stays below this ceiling. Only needed when a container
|
||||
# legitimately has large stable data. Growth detection covers everything else.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# STORAGE_WATCHDOG_STATE_FILE — strike counts (/tmp — resets on reboot ✅)
|
||||
# WATCHDOG_APPDATA_GROWTH_FILE — per-container size baseline (/tmp — resets on reboot ✅)
|
||||
#
|
||||
# /tmp files reset on reboot — correct. Pre-reboot strikes and growth baselines are
|
||||
# meaningless after a reboot. Both rebuild cleanly from cycle 1.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# storage_watchdog.sh
|
||||
# Single-pass storage health check. Silent if all healthy.
|
||||
#
|
||||
# storage_watchdog.sh --dry-run
|
||||
# Run all checks without truncating anything. Shows what would be actioned.
|
||||
#
|
||||
# storage_watchdog.sh --status
|
||||
# Show configuration, active strikes, and growth baseline status. Then exit.
|
||||
#
|
||||
# storage_watchdog.sh --log
|
||||
# Verbose output — every container checked, every size comparison, every decision.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be truncated"
|
||||
|
||||
touch "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
|
||||
touch "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STORAGE WATCHDOG STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Appdata check: ${WATCHDOG_CHECK_APPDATA:-true}"
|
||||
echo "$ICON_GEAR Paths: ${WATCHDOG_APPDATA_PATHS[*]:-none}"
|
||||
echo "$ICON_GEAR Growth thresh: ${WATCHDOG_APPDATA_GROWTH_GB:-2}GB/cycle"
|
||||
echo "$ICON_GEAR Log max: ${WATCHDOG_APPDATA_LOG_MAX_GB:-2}GB"
|
||||
echo "$ICON_GEAR Truncate logs: ${WATCHDOG_APPDATA_TRUNCATE_LOGS:-false}"
|
||||
echo "$ICON_GEAR Strike limit: ${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}"
|
||||
echo ""
|
||||
echo "── Active Strikes ──"
|
||||
if [[ -s "$STORAGE_WATCHDOG_STATE_FILE" ]]; then
|
||||
while IFS=':' read -r _sk _sv; do
|
||||
_sv_clean="${_sv//[^0-9]/}"
|
||||
[[ "${_sv_clean:-0}" -gt 0 ]] && echo " $_sk → $_sv_clean strikes"
|
||||
done < "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
else
|
||||
echo " none"
|
||||
fi
|
||||
echo ""
|
||||
echo "── Growth Baseline ──"
|
||||
if [[ -s "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
|
||||
_entries=$(wc -l < "$WATCHDOG_APPDATA_GROWTH_FILE")
|
||||
_age=$(( $(date +%s) - $(stat -c %Y "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null || echo 0) ))
|
||||
echo " Entries: $_entries containers Age: $(( _age / 60 ))m ago"
|
||||
else
|
||||
echo " No baseline yet (builds on first cycle after boot)"
|
||||
fi
|
||||
echo ""
|
||||
echo "── Suppress Ceilings (this host) ──"
|
||||
if [[ ${#WATCHDOG_APPDATA_SIZES[@]} -gt 0 ]]; then
|
||||
for _c in "${!WATCHDOG_APPDATA_SIZES[@]}"; do
|
||||
_ceil_gb=$(awk "BEGIN {printf \"%.0f\", ${WATCHDOG_APPDATA_SIZES[$_c]} / 1024}")
|
||||
echo " $_c → ${_ceil_gb}GB"
|
||||
done
|
||||
else
|
||||
echo " none configured"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Appdata Size Monitoring ━━━
|
||||
# ==============================================================================================
|
||||
[[ "$WATCHDOG_CHECK_APPDATA" != "true" ]] && exit 0
|
||||
|
||||
log "$ICON_GEAR Storage watchdog — $MY_ID — $(date '+%H:%M:%S')"
|
||||
|
||||
WARNINGS=0
|
||||
|
||||
_STRIKE_LIMIT=${WATCHDOG_APPDATA_STRIKE_LIMIT:-3}
|
||||
_GROWTH_MB=$(( ${WATCHDOG_APPDATA_GROWTH_GB:-2} * 1024 ))
|
||||
_LOG_KB=$(( ${WATCHDOG_APPDATA_LOG_MAX_GB:-2} * 1024 * 1024 ))
|
||||
|
||||
# Tracks files handled by Part 1 to prevent duplicate alerts in Part 2
|
||||
declare -A _HANDLED=()
|
||||
|
||||
for _appdata_path in "${WATCHDOG_APPDATA_PATHS[@]:-}"; do
|
||||
[[ -z "$_appdata_path" || ! -d "$_appdata_path" ]] && continue
|
||||
|
||||
# ── Part 1: Growth rate scan ──────────────────────────────────────────────────────────────
|
||||
declare -A _PREV=()
|
||||
if [[ -f "$WATCHDOG_APPDATA_GROWTH_FILE" ]]; then
|
||||
while IFS='|' read -r _cn _cs _; do
|
||||
[[ -n "$_cn" ]] && _PREV["$_cn"]="$_cs"
|
||||
done < "$WATCHDOG_APPDATA_GROWTH_FILE"
|
||||
fi
|
||||
|
||||
_growth_tmp=$(mktemp 2>/dev/null) || _growth_tmp=""
|
||||
_now=$(date +%s)
|
||||
|
||||
while IFS= read -r _du_line; do
|
||||
_curr_mb=$(echo "$_du_line" | awk '{print $1}')
|
||||
_cdir=$(echo "$_du_line" | awk '{print $2}')
|
||||
_cname=$(basename "$_cdir")
|
||||
[[ -z "$_cname" || "$_cname" == "*" ]] && continue
|
||||
|
||||
[[ -n "$_growth_tmp" ]] && echo "${_cname}|${_curr_mb}|${_now}" >> "$_growth_tmp"
|
||||
|
||||
_prev_mb="${_PREV[$_cname]:-}"
|
||||
[[ -z "$_prev_mb" ]] && continue # First run after boot — building baseline
|
||||
|
||||
_growth_mb=$(( _curr_mb - _prev_mb ))
|
||||
_safe=$(echo "$_cname" | tr -cd '[:alnum:]_')
|
||||
|
||||
# Condition resolved — growth stopped, clear strikes
|
||||
if [[ "$_growth_mb" -le 0 ]]; then
|
||||
_existing=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
|
||||
_existing="${_existing//[^0-9]/}"
|
||||
[[ "${_existing:-0}" -gt 0 ]] && \
|
||||
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Check suppress ceiling
|
||||
_ceiling="${WATCHDOG_APPDATA_SIZES[$_cname]:-}"
|
||||
if [[ -n "$_ceiling" && "$_curr_mb" -lt "$_ceiling" ]]; then
|
||||
log "$_cname — growth suppressed (${_curr_mb}MB < ${_ceiling}MB ceiling)"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Growth exceeds threshold — strike logic
|
||||
if [[ "$_growth_mb" -ge "$_GROWTH_MB" ]]; then
|
||||
_growth_gb=$(awk "BEGIN {printf \"%.1f\", $_growth_mb / 1024}")
|
||||
_curr_gb=$(awk "BEGIN {printf \"%.1f\", $_curr_mb / 1024}")
|
||||
|
||||
_strikes=$(get_strikes "appdata_growth_${_safe}" "$STORAGE_WATCHDOG_STATE_FILE")
|
||||
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
|
||||
|
||||
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
|
||||
_strikes=$(( _strikes + 1 ))
|
||||
set_strikes "appdata_growth_${_safe}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
}
|
||||
|
||||
warn "$_cname — grew ${_growth_gb}GB this cycle (total: ${_curr_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
|
||||
(( WARNINGS++ ))
|
||||
|
||||
# Focused *.log scan inside the growing container's dir
|
||||
_found_logs=()
|
||||
while IFS= read -r _lf; do
|
||||
[[ -n "$_lf" ]] && _found_logs+=("$_lf")
|
||||
done < <(find "$_cdir" -maxdepth 3 -type f \
|
||||
\( -name "*.log" -o -name "*.log.*" \) \
|
||||
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
|
||||
|
||||
_log_summary=""
|
||||
[[ ${#_found_logs[@]} -gt 0 ]] && _log_summary=$(printf '%s\n' "${_found_logs[@]}" | \
|
||||
awk '{printf "%.1fGB %s | ", $1/1073741824, $2}' | head -c 200)
|
||||
|
||||
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
|
||||
if [[ -n "$_log_summary" ]]; then
|
||||
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — logs: ${_log_summary}" \
|
||||
"Storage Watchdog" "warning"
|
||||
else
|
||||
notify "$_cname grew ${_growth_gb}GB on $(hostname) [strike ${_strikes}/${_STRIKE_LIMIT}] — data growth, no log files" \
|
||||
"Storage Watchdog" "warning"
|
||||
fi
|
||||
else
|
||||
# Action cycle
|
||||
error "$_cname — growth strike limit reached (${_STRIKE_LIMIT} consecutive cycles, ${_growth_gb}GB this cycle)"
|
||||
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" && ${#_found_logs[@]} -gt 0 ]]; then
|
||||
for _lf_entry in "${_found_logs[@]}"; do
|
||||
_lf_path=$(echo "$_lf_entry" | cut -d' ' -f2-)
|
||||
_lf_gb=$(echo "$_lf_entry" | awk '{printf "%.1f", $1/1073741824}')
|
||||
_HANDLED["$_lf_path"]=1
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would truncate $_lf_path (${_lf_gb}GB)"
|
||||
else
|
||||
if truncate -s 0 "$_lf_path" 2>/dev/null; then
|
||||
success "Truncated runaway log: $_lf_path (was ${_lf_gb}GB)"
|
||||
notify "Truncated runaway log on $(hostname): $_lf_path (was ${_lf_gb}GB)" \
|
||||
"Storage Watchdog" "warning"
|
||||
else
|
||||
warn "Failed to truncate $_lf_path"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
set_strikes "appdata_growth_${_safe}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
else
|
||||
if [[ -n "$_log_summary" ]]; then
|
||||
notify "$_cname appdata runaway on $(hostname) — ${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — logs: ${_log_summary}" \
|
||||
"Storage Watchdog" "critical"
|
||||
else
|
||||
notify "$_cname appdata runaway on $(hostname) — ${_growth_gb}GB growth for ${_STRIKE_LIMIT} cycles — data growth, manual investigation needed" \
|
||||
"Storage Watchdog" "critical"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# Mark found logs as handled to suppress Part 2 duplicates this cycle
|
||||
for _lf_entry in "${_found_logs[@]}"; do
|
||||
_HANDLED["$(echo "$_lf_entry" | cut -d' ' -f2-)"]=1
|
||||
done
|
||||
fi
|
||||
done < <(du -sm "$_appdata_path"/*/ 2>/dev/null)
|
||||
|
||||
# Atomically update growth baseline
|
||||
[[ -n "$_growth_tmp" ]] && mv "$_growth_tmp" "$WATCHDOG_APPDATA_GROWTH_FILE" 2>/dev/null
|
||||
|
||||
# ── Part 2: Absolute log size scan ────────────────────────────────────────────────────────
|
||||
# Catches *.log files already large but no longer actively growing this cycle.
|
||||
# Same strike logic. Skips files already handled by Part 1 above.
|
||||
declare -A _LOG_SEEN=()
|
||||
|
||||
while IFS= read -r _hit; do
|
||||
[[ -z "$_hit" ]] && continue
|
||||
_fpath=$(echo "$_hit" | cut -d' ' -f2-)
|
||||
[[ -n "${_HANDLED[$_fpath]:-}" ]] && continue
|
||||
|
||||
_fsize_bytes=$(echo "$_hit" | awk '{print $1}')
|
||||
_fsize_gb=$(awk "BEGIN {printf \"%.1f\", $_fsize_bytes / 1073741824}")
|
||||
_safe_fkey=$(echo "$_fpath" | tr -cd '[:alnum:]_' | cut -c1-120)
|
||||
_LOG_SEEN["appdata_log_${_safe_fkey}"]=1
|
||||
|
||||
_strikes=$(get_strikes "appdata_log_${_safe_fkey}" "$STORAGE_WATCHDOG_STATE_FILE")
|
||||
_strikes="${_strikes//[^0-9]/}"; _strikes="${_strikes:-0}"
|
||||
|
||||
[[ "$_strikes" -lt "$_STRIKE_LIMIT" ]] && {
|
||||
_strikes=$(( _strikes + 1 ))
|
||||
set_strikes "appdata_log_${_safe_fkey}" "$_strikes" "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
}
|
||||
|
||||
warn "Oversized log: $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]"
|
||||
(( WARNINGS++ ))
|
||||
|
||||
if [[ "$_strikes" -lt "$_STRIKE_LIMIT" ]]; then
|
||||
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) [strike ${_strikes}/${_STRIKE_LIMIT}]" \
|
||||
"Storage Watchdog" "warning"
|
||||
else
|
||||
error "Oversized log persists for ${_STRIKE_LIMIT} cycles: $_fpath (${_fsize_gb}GB)"
|
||||
if [[ "$WATCHDOG_APPDATA_TRUNCATE_LOGS" == "true" ]]; then
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would truncate $_fpath (${_fsize_gb}GB)"
|
||||
else
|
||||
if truncate -s 0 "$_fpath" 2>/dev/null; then
|
||||
success "Truncated oversized log: $_fpath (was ${_fsize_gb}GB)"
|
||||
notify "Truncated oversized log on $(hostname): $_fpath (was ${_fsize_gb}GB)" \
|
||||
"Storage Watchdog" "warning"
|
||||
set_strikes "appdata_log_${_safe_fkey}" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
unset "_LOG_SEEN[appdata_log_${_safe_fkey}]"
|
||||
else
|
||||
warn "Failed to truncate $_fpath"
|
||||
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, truncate failed" \
|
||||
"Storage Watchdog" "critical"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
notify "Oversized log on $(hostname): $_fpath (${_fsize_gb}GB) — ${_STRIKE_LIMIT} cycles, manual intervention needed" \
|
||||
"Storage Watchdog" "critical"
|
||||
fi
|
||||
fi
|
||||
done < <(find "$_appdata_path" -maxdepth 4 -type f \
|
||||
\( -name "*.log" -o -name "*.log.*" \) \
|
||||
-size +${_LOG_KB}k -printf "%s %p\n" 2>/dev/null)
|
||||
|
||||
# Auto-clear strikes for log files no longer oversized this cycle
|
||||
while IFS=':' read -r _sk _sv; do
|
||||
[[ "$_sk" != appdata_log_* ]] && continue
|
||||
_sv_clean="${_sv//[^0-9]/}"
|
||||
[[ "${_sv_clean:-0}" -eq 0 ]] && continue
|
||||
[[ -n "${_LOG_SEEN[$_sk]:-}" ]] && continue
|
||||
set_strikes "$_sk" 0 "$STORAGE_WATCHDOG_STATE_FILE"
|
||||
done < "$STORAGE_WATCHDOG_STATE_FILE" 2>/dev/null
|
||||
|
||||
unset _LOG_SEEN
|
||||
|
||||
done
|
||||
|
||||
unset _PREV _HANDLED
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$WARNINGS" -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Storage Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_WATCHDOG Warnings: $WARNINGS"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
else
|
||||
log "Storage healthy ✅ ($(date '+%H:%M:%S'))"
|
||||
fi
|
||||
Executable
+805
@@ -0,0 +1,805 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= System Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Last line of defense — reboots the system cleanly if it is about to become
|
||||
# unstable. Runs continuously as a background process started by
|
||||
# array_started.sh at array start. Works alongside docker_watchdog.sh which
|
||||
# handles container-level healing first. Only escalates to reboot when
|
||||
# docker_watchdog.sh cannot resolve the condition.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three-Tier Response System
|
||||
#
|
||||
# Tier 1 — CRITICAL (bypass all strikes, reboot immediately)
|
||||
# Docker daemon unresponsive — nothing can be healed; running it longer makes it worse
|
||||
# rootfs at 99%+ — writes failing; SSH may stop; no recovery options
|
||||
# Kernel oops/BUG in dmesg — kernel running with corrupted state
|
||||
# File descriptor exhaustion — new connections and processes failing silently
|
||||
# /boot read-only unexpectedly — state files and config writes silently failing
|
||||
#
|
||||
# Tier 2 — URGENT (bypass strikes when OOM confirms active crisis)
|
||||
# RAM < MEM_GB AND OOM kills >= OOM_LIMIT in this cycle.
|
||||
# OOM kills at this rate means the system is dying faster than watchdogs can heal.
|
||||
# Without OOM confirmation → standard strike system applies.
|
||||
#
|
||||
# Tier 3 — STANDARD (N consecutive failures → reboot)
|
||||
# RAM tiers, load, CPU temp, zombies, /var/log, /tmp, containers, NIC, mdstat.
|
||||
#
|
||||
# RAM Tiers
|
||||
# MEM_WARN_GB (10GB) — warn + notify only
|
||||
# MEM_SHUTDOWN_GB (6GB) — stop non-essential containers, wait for recovery
|
||||
# MEM_GB (4GB) — strike system → reboot (bypass with OOM confirmation)
|
||||
# MEM_RECOVER_GB (30GB) — RAM must reach this before stopped containers restart
|
||||
#
|
||||
# Container Shutdown Logic (at MEM_SHUTDOWN_GB)
|
||||
# Stops all containers not in SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED.
|
||||
# Stopped containers tracked in shutdown list — won't restart until RAM recovers.
|
||||
# Strike system prevents flip-flopping — shutdown only once per degradation event.
|
||||
#
|
||||
# Abort Conditions (prevent reboot during sensitive operations)
|
||||
# ZFS pool unhealthy, parity running, mover running — each toggleable.
|
||||
# CRITICAL tier bypasses all abort conditions — imminent crash overrides data safety.
|
||||
#
|
||||
# Checks Run Every Cycle
|
||||
# rootfs usage, /var/log, /tmp, free RAM, ZFS ARC, CPU temp, load avg,
|
||||
# zombie processes, Docker daemon, OOM rate, /boot read-only, kernel oops,
|
||||
# file descriptor exhaustion, array disk errors, NIC state, required containers.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# Reboot and container stop require root.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents a second watchdog instance from starting.
|
||||
#
|
||||
# State File Verification
|
||||
# All state files verified writable at startup — errors if any cannot be created.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf — System Watchdog section
|
||||
# Full variable listing in master.conf. Key variables:
|
||||
#
|
||||
# SYS_WATCHDOG_REBOOT_WINDOW_HRS — reboot rate limit window (default: 2)
|
||||
# SYS_WATCHDOG_MAX_REBOOTS — max reboots in window before giving up (default: 3)
|
||||
# SYS_WATCHDOG_STRIKES — consecutive failures before reboot (default: 3)
|
||||
# SYS_WATCHDOG_OOM_LIMIT — OOM kills/cycle to trigger URGENT bypass (default: 3)
|
||||
# SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED — containers exempt from memory shutdown
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# SYS_WATCHDOG_STATE_FILE — strike counters and cycle state
|
||||
# SYS_WATCHDOG_REBOOT_LOG — reboot history for rate limiting
|
||||
# SYS_WATCHDOG_FAILED_FILE — containers confirmed down for skip list integration
|
||||
# SYS_WATCHDOG_OOM_FILE — OOM kill counter from previous cycle
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# system_watchdog.sh
|
||||
# Start continuous monitoring loop. Runs until stopped or system reboots.
|
||||
#
|
||||
# system_watchdog.sh --dry-run
|
||||
# Run detection logic without rebooting or stopping containers.
|
||||
#
|
||||
# system_watchdog.sh --status
|
||||
# Show config, thresholds, current system state, and strike counts.
|
||||
#
|
||||
# system_watchdog.sh --log
|
||||
# Verbose per-cycle output — show every check result and threshold comparison.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup — runs once at start ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
validate_unraid_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
TOTAL_CORES=$(nproc)
|
||||
DOCKER_TIMEOUT=10
|
||||
SYS_WATCHDOG_REBOOT_WINDOW=$(( SYS_WATCHDOG_REBOOT_WINDOW_HRS * 3600 ))
|
||||
|
||||
# Ensure state files exist
|
||||
for state_file in "$SYS_WATCHDOG_STATE_FILE" "$SYS_WATCHDOG_REBOOT_LOG" \
|
||||
"$SYS_WATCHDOG_FAILED_FILE" "$SYS_WATCHDOG_OOM_FILE"; do
|
||||
touch "$state_file" 2>/dev/null || {
|
||||
error "Cannot create state file: $state_file"
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no reboots or container shutdowns will occur"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SYSTEM WATCHDOG STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
echo "── Tier 1 — CRITICAL (bypass strikes immediately) ──"
|
||||
echo "$ICON_DISK rootfs critical: ${SYS_WATCHDOG_ROOTFS_CRITICAL_PCT}%"
|
||||
echo "$ICON_GEAR FD critical: ${SYS_WATCHDOG_FD_CRITICAL_PCT}%"
|
||||
echo "$ICON_GEAR /boot read-only: check=${SYS_WATCHDOG_CHECK_BOOT}"
|
||||
echo "$ICON_GEAR Kernel oops: check=${SYS_WATCHDOG_CHECK_KERNEL_OOPS}"
|
||||
echo "$ICON_CONTAINERS Docker daemon: check=${SYS_WATCHDOG_CHECK_DOCKER_DAEMON}"
|
||||
echo ""
|
||||
echo "── Tier 2 — URGENT (bypass strikes with OOM confirmation) ──"
|
||||
echo "$ICON_MEM RAM critical: < ${SYS_WATCHDOG_MEM_GB}GB"
|
||||
echo "$ICON_GEAR OOM limit: ${SYS_WATCHDOG_OOM_LIMIT} kills/cycle"
|
||||
echo ""
|
||||
echo "── Tier 3 — STANDARD (strike system) ──"
|
||||
echo "$ICON_DISK rootfs warn: ${SYS_WATCHDOG_ROOTFS_PCT}%"
|
||||
echo "$ICON_GEAR /var/log warn: ${SYS_WATCHDOG_LOG_PCT}%"
|
||||
echo "$ICON_GEAR /tmp warn: ${SYS_WATCHDOG_TMP_PCT}%"
|
||||
echo "$ICON_MEM RAM reboot: < ${SYS_WATCHDOG_MEM_GB}GB (+ strikes)"
|
||||
echo " (RAM warn/shutdown/recover managed by resource_watchdog.sh)"
|
||||
echo "$ICON_ZFS ARC pinned: ${SYS_WATCHDOG_ARC_PINNED_PCT}%"
|
||||
echo "$ICON_GEAR Load multiplier: ${SYS_WATCHDOG_LOAD_MULTIPLIER}x (= $(( TOTAL_CORES * SYS_WATCHDOG_LOAD_MULTIPLIER )) on $TOTAL_CORES cores)"
|
||||
echo "$ICON_GEAR Zombie limit: ${SYS_WATCHDOG_ZOMBIE_LIMIT}"
|
||||
echo "$ICON_GEAR CPU temp max: ${SYS_WATCHDOG_CPU_TEMP_MAX}°C"
|
||||
echo "$ICON_GEAR Strike limit: ${SYS_WATCHDOG_STRIKE_LIMIT} cycles"
|
||||
echo "$ICON_TIME Interval: ${SYSTEM_WATCHDOG_INTERVAL}s"
|
||||
echo "$ICON_REBOOT_SMART Reboot limit: ${SYS_WATCHDOG_REBOOT_LIMIT} in ${SYS_WATCHDOG_REBOOT_WINDOW_HRS}hr"
|
||||
echo ""
|
||||
echo ""
|
||||
echo "── Check Toggles ──"
|
||||
echo " rootfs=$SYS_WATCHDOG_CHECK_ROOTFS log=$SYS_WATCHDOG_CHECK_LOG ram=$SYS_WATCHDOG_CHECK_RAM"
|
||||
echo " arc=$SYS_WATCHDOG_CHECK_ARC cpu_temp=$SYS_WATCHDOG_CHECK_CPU_TEMP load=$SYS_WATCHDOG_CHECK_LOAD"
|
||||
echo " zombies=$SYS_WATCHDOG_CHECK_ZOMBIES docker=$SYS_WATCHDOG_CHECK_DOCKER_DAEMON"
|
||||
echo " containers=$SYS_WATCHDOG_CHECK_CONTAINERS oom=$SYS_WATCHDOG_CHECK_OOM"
|
||||
echo " tmp=$SYS_WATCHDOG_CHECK_TMP fd=$SYS_WATCHDOG_CHECK_FD boot=$SYS_WATCHDOG_CHECK_BOOT"
|
||||
echo " kernel_oops=$SYS_WATCHDOG_CHECK_KERNEL_OOPS sshd=$SYS_WATCHDOG_CHECK_SSHD"
|
||||
echo " network=$SYS_WATCHDOG_CHECK_NETWORK mdstat=$SYS_WATCHDOG_CHECK_MDSTAT"
|
||||
echo " runaway=$SYS_WATCHDOG_CHECK_RUNAWAY"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── STATE HELPERS ─────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
get_strikes() {
|
||||
grep -E "^${1}:" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | cut -d':' -f2
|
||||
}
|
||||
|
||||
set_strikes() {
|
||||
local key="$1" count="$2"
|
||||
grep -vE "^${key}:" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null > "${SYS_WATCHDOG_STATE_FILE}.tmp"
|
||||
echo "${key}:${count}" >> "${SYS_WATCHDOG_STATE_FILE}.tmp"
|
||||
mv "${SYS_WATCHDOG_STATE_FILE}.tmp" "$SYS_WATCHDOG_STATE_FILE"
|
||||
}
|
||||
|
||||
increment_strikes() {
|
||||
local key="$1"
|
||||
local current
|
||||
current=$(get_strikes "$key")
|
||||
[[ -z "$current" ]] && current=0
|
||||
(( current++ ))
|
||||
set_strikes "$key" "$current"
|
||||
echo "$current"
|
||||
}
|
||||
|
||||
reset_strikes() {
|
||||
set_strikes "$1" 0
|
||||
}
|
||||
|
||||
get_state_val() {
|
||||
grep -E "^${1}=" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | cut -d'=' -f2
|
||||
}
|
||||
|
||||
set_state_val() {
|
||||
local key="$1" val="$2"
|
||||
grep -vE "^${key}=" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null > "${SYS_WATCHDOG_STATE_FILE}.tmp"
|
||||
echo "${key}=${val}" >> "${SYS_WATCHDOG_STATE_FILE}.tmp"
|
||||
mv "${SYS_WATCHDOG_STATE_FILE}.tmp" "$SYS_WATCHDOG_STATE_FILE"
|
||||
}
|
||||
|
||||
purge_old_reboots() {
|
||||
local now cutoff
|
||||
now=$(date +%s)
|
||||
cutoff=$(( now - SYS_WATCHDOG_REBOOT_WINDOW ))
|
||||
grep -v "^$" "$SYS_WATCHDOG_REBOOT_LOG" 2>/dev/null | while IFS= read -r ts; do
|
||||
[[ "$ts" -gt "$cutoff" ]] && echo "$ts"
|
||||
done > "${SYS_WATCHDOG_REBOOT_LOG}.tmp"
|
||||
mv "${SYS_WATCHDOG_REBOOT_LOG}.tmp" "$SYS_WATCHDOG_REBOOT_LOG"
|
||||
}
|
||||
|
||||
count_recent_reboots() {
|
||||
purge_old_reboots
|
||||
grep -c "." "$SYS_WATCHDOG_REBOOT_LOG" 2>/dev/null || echo 0
|
||||
}
|
||||
|
||||
log_reboot() {
|
||||
date +%s >> "$SYS_WATCHDOG_REBOOT_LOG"
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── OOM TRACKING ──────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Reads /proc/vmstat oom_kill counter — delta per cycle = rate of OOM kills
|
||||
# Used for Tier 2 bypass and diagnostic context in reboot messages
|
||||
|
||||
get_oom_delta() {
|
||||
local current_oom
|
||||
current_oom=$(grep "^oom_kill " /proc/vmstat 2>/dev/null | awk '{print $2}')
|
||||
[[ -z "$current_oom" ]] && echo 0 && return
|
||||
|
||||
local prev_oom
|
||||
prev_oom=$(cat "$SYS_WATCHDOG_OOM_FILE" 2>/dev/null || echo 0)
|
||||
echo "$current_oom" > "$SYS_WATCHDOG_OOM_FILE"
|
||||
|
||||
local delta=$(( current_oom - prev_oom ))
|
||||
[[ "$delta" -lt 0 ]] && delta=0 # counter reset on reboot
|
||||
echo "$delta"
|
||||
}
|
||||
|
||||
get_oom_victims() {
|
||||
# Get process names from dmesg that were OOM killed this boot
|
||||
dmesg -T 2>/dev/null | grep -i "Killed process" | \
|
||||
awk '{print $NF}' | sort | uniq -c | sort -rn | head -5 | \
|
||||
awk '{printf "%s×%d ", $2, $1}' | sed 's/ $//'
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── ABORT CONDITIONS ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Returns 1 if reboot should be aborted, 0 if reboot should proceed
|
||||
# CRITICAL tier bypasses this function entirely
|
||||
|
||||
check_abort_conditions() {
|
||||
local should_abort=false
|
||||
|
||||
if command -v zpool >/dev/null 2>&1; then
|
||||
local unhealthy
|
||||
unhealthy=$(zpool list -H -o health 2>/dev/null | grep -v ONLINE || true)
|
||||
if [[ -n "$unhealthy" ]]; then
|
||||
if [[ "$SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY" == true ]]; then
|
||||
error "ZFS pool unhealthy — aborting reboot to prevent data loss"
|
||||
notify "System watchdog aborted reboot on $(hostname) ($MY_ID) — ZFS pool unhealthy" \
|
||||
"System Watchdog" "warning"
|
||||
should_abort=true
|
||||
else
|
||||
warn "ZFS pool unhealthy — continuing reboot (ABORT_ON_ZFS_UNHEALTHY=false)"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
if grep -q "progress" /var/local/emhttp/parity-date.txt 2>/dev/null; then
|
||||
if [[ "$SYS_WATCHDOG_ABORT_ON_PARITY" == true ]]; then
|
||||
error "Parity check running — aborting reboot"
|
||||
notify "System watchdog aborted reboot on $(hostname) ($MY_ID) — parity running" \
|
||||
"System Watchdog" "warning"
|
||||
should_abort=true
|
||||
else
|
||||
warn "Parity check running — continuing reboot (ABORT_ON_PARITY=false)"
|
||||
fi
|
||||
fi
|
||||
|
||||
if pgrep -f "mover" >/dev/null 2>&1; then
|
||||
if [[ "$SYS_WATCHDOG_ABORT_ON_MOVER" == true ]]; then
|
||||
error "Mover running — aborting reboot"
|
||||
notify "System watchdog aborted reboot on $(hostname) ($MY_ID) — mover running" \
|
||||
"System Watchdog" "warning"
|
||||
should_abort=true
|
||||
else
|
||||
warn "Mover running — continuing reboot (ABORT_ON_MOVER=false)"
|
||||
fi
|
||||
fi
|
||||
|
||||
[[ "$should_abort" == true ]] && return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── STANDARD STRIKE CHECK ─────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Returns 0 = reboot now | 1 = not yet
|
||||
|
||||
run_strike_check() {
|
||||
local key="$1" triggered="$2" description="$3"
|
||||
if [[ "$triggered" == true ]]; then
|
||||
local strikes
|
||||
strikes=$(increment_strikes "$key")
|
||||
warn "$description — strike $strikes/$SYS_WATCHDOG_STRIKE_LIMIT"
|
||||
if (( strikes >= SYS_WATCHDOG_STRIKE_LIMIT )); then
|
||||
error "$description — strike limit hit, reboot triggered"
|
||||
reset_strikes "$key"
|
||||
return 0
|
||||
fi
|
||||
else
|
||||
local current
|
||||
current=$(get_strikes "$key")
|
||||
[[ -n "$current" && "$current" -gt 0 ]] && reset_strikes "$key"
|
||||
fi
|
||||
return 1
|
||||
}
|
||||
|
||||
# ── Exit Trap — restart containers stopped before an aborted reboot ───────────────────────────
|
||||
_SYS_REBOOT_STOPPED=()
|
||||
_trap_sys_reboot_restart() {
|
||||
[[ ${#_SYS_REBOOT_STOPPED[@]} -eq 0 ]] && return
|
||||
warn "Exit trap: restarting containers stopped before aborted reboot"
|
||||
for c in "${_SYS_REBOOT_STOPPED[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
docker inspect "$c" >/dev/null 2>&1 && docker start "$c" >/dev/null 2>&1 || true
|
||||
done
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ── DO REBOOT ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# tier: "critical" (bypass abort) | "urgent" | "standard"
|
||||
|
||||
do_reboot() {
|
||||
local tier="${1:-standard}"
|
||||
shift
|
||||
local triggers=("$@")
|
||||
|
||||
# Get OOM context for reboot message
|
||||
local oom_victims=""
|
||||
if [[ "$SYS_WATCHDOG_CHECK_OOM" == true ]]; then
|
||||
oom_victims=$(get_oom_victims)
|
||||
[[ -n "$oom_victims" ]] && triggers+=("oom_victims: $oom_victims")
|
||||
fi
|
||||
|
||||
# Abort check — CRITICAL bypasses this
|
||||
if [[ "$tier" != "critical" ]]; then
|
||||
if ! check_abort_conditions; then
|
||||
return
|
||||
fi
|
||||
else
|
||||
warn "CRITICAL tier — bypassing abort conditions"
|
||||
fi
|
||||
|
||||
RECENT_REBOOTS=$(count_recent_reboots)
|
||||
log "Recent reboots in ${SYS_WATCHDOG_REBOOT_WINDOW_HRS}hr window: $RECENT_REBOOTS / $SYS_WATCHDOG_REBOOT_LIMIT"
|
||||
|
||||
if [[ "$RECENT_REBOOTS" -ge "$SYS_WATCHDOG_REBOOT_LIMIT" ]]; then
|
||||
error "Reboot loop detected — shutting down instead of rebooting"
|
||||
notify "Reboot loop on $(hostname) ($MY_ID) — shutting down after $RECENT_REBOOTS reboots — ${triggers[*]}" \
|
||||
"System Watchdog" "warning"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would shutdown now"
|
||||
return
|
||||
fi
|
||||
sync
|
||||
/sbin/poweroff
|
||||
return
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " $ICON_REBOOT_SMART SYSTEM WATCHDOG — REBOOT TRIGGERED"
|
||||
echo " Tier: ${tier^^}"
|
||||
echo " Host: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
for t in "${triggers[@]}"; do
|
||||
echo " → $t"
|
||||
done
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
notify "System watchdog ${tier^^} reboot on $(hostname) ($MY_ID) — ${triggers[*]}" \
|
||||
"System Watchdog" "warning"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — reboot sequence would begin now"
|
||||
return
|
||||
fi
|
||||
|
||||
log_reboot
|
||||
|
||||
# Graceful shutdown sequence
|
||||
warn "Shutting down VMs..."
|
||||
if command -v virsh >/dev/null 2>&1; then
|
||||
for VM in $(virsh list --name 2>/dev/null); do
|
||||
[[ -z "$VM" ]] && continue
|
||||
virsh shutdown "$VM" >/dev/null 2>&1
|
||||
done
|
||||
sleep 30
|
||||
fi
|
||||
|
||||
warn "Stopping Docker containers..."
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
mapfile -t _SYS_REBOOT_STOPPED < <(docker ps --format '{{.Names}}' 2>/dev/null)
|
||||
trap _trap_sys_reboot_restart EXIT
|
||||
timeout 60 docker ps -q 2>/dev/null | xargs -r docker stop >/dev/null 2>&1
|
||||
fi
|
||||
|
||||
warn "Stopping User Scripts..."
|
||||
pkill -f "/tmp/user.scripts" 2>/dev/null || true
|
||||
|
||||
warn "Syncing disks..."
|
||||
sync
|
||||
|
||||
trap - EXIT # committed to reboot — containers should stay down
|
||||
sleep 5
|
||||
/sbin/reboot
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Single-Pass Health Check ━━━
|
||||
# ==============================================================================================
|
||||
warn "System watchdog — $MY_ID — $(date '+%H:%M:%S')"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
TRIGGERS=()
|
||||
CRITICAL_TRIGGERS=()
|
||||
URGENT_OOM_CONFIRMED=false
|
||||
|
||||
# ── OOM Delta — read every cycle for bypass decisions ─────────────────────────────────────
|
||||
OOM_DELTA=0
|
||||
if [[ "$SYS_WATCHDOG_CHECK_OOM" == true ]]; then
|
||||
OOM_DELTA=$(get_oom_delta)
|
||||
[[ "$OOM_DELTA" -gt 0 ]] && \
|
||||
log "OOM kills this cycle: $OOM_DELTA (limit: ${SYS_WATCHDOG_OOM_LIMIT})"
|
||||
fi
|
||||
|
||||
# ==========================================================================================
|
||||
# ━━━ TIER 1 — CRITICAL CHECKS (bypass all strikes, reboot immediately) ━━━
|
||||
# ==========================================================================================
|
||||
|
||||
# ── Docker daemon — critical: nothing can heal without it ─────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_DOCKER_DAEMON" == true ]]; then
|
||||
if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
|
||||
error "Docker daemon unresponsive — CRITICAL"
|
||||
|
||||
# Attempt daemon restart before rebooting
|
||||
warn "Attempting Docker daemon restart..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
/etc/rc.d/rc.docker restart >/dev/null 2>&1
|
||||
sleep 15
|
||||
if timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
|
||||
warn "Docker daemon restarted successfully — continuing monitoring"
|
||||
else
|
||||
error "Docker daemon restart failed — adding to CRITICAL triggers"
|
||||
CRITICAL_TRIGGERS+=("docker_daemon_unresponsive")
|
||||
fi
|
||||
else
|
||||
warn "DRY RUN — would attempt Docker daemon restart"
|
||||
CRITICAL_TRIGGERS+=("docker_daemon_unresponsive")
|
||||
fi
|
||||
else
|
||||
log "Docker daemon healthy ✅"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── rootfs critical — at 99%+ writes are failing ─────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_ROOTFS" == true ]]; then
|
||||
ROOTFS_USED=$(df / --output=pcent 2>/dev/null | tail -1 | tr -d ' %')
|
||||
if [[ "$ROOTFS_USED" -ge "${SYS_WATCHDOG_ROOTFS_CRITICAL_PCT:-99}" ]]; then
|
||||
error "rootfs ${ROOTFS_USED}% — CRITICAL (writes failing)"
|
||||
CRITICAL_TRIGGERS+=("rootfs_full=${ROOTFS_USED}%")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Kernel oops/BUG — kernel running with corrupted state ────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_KERNEL_OOPS" == true ]]; then
|
||||
PREV_OOPS=$(get_state_val "kernel_oops_count")
|
||||
CURRENT_OOPS=$(dmesg 2>/dev/null | grep -cE "BUG:|kernel BUG|Oops:" || echo 0)
|
||||
CURRENT_OOPS="${CURRENT_OOPS//[^0-9]/}"; CURRENT_OOPS="${CURRENT_OOPS:-0}"
|
||||
set_state_val "kernel_oops_count" "$CURRENT_OOPS"
|
||||
|
||||
if [[ -n "$PREV_OOPS" && "$PREV_OOPS" =~ ^[0-9]+$ ]]; then
|
||||
OOPS_DELTA=$(( CURRENT_OOPS - PREV_OOPS ))
|
||||
if [[ "$OOPS_DELTA" -gt 0 ]]; then
|
||||
error "Kernel oops/BUG detected — $OOPS_DELTA new since last cycle — CRITICAL"
|
||||
CRITICAL_TRIGGERS+=("kernel_oops=${OOPS_DELTA}_new")
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── File descriptor exhaustion — new connections failing silently ─────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_FD" == true ]]; then
|
||||
FD_LINE=$(cat /proc/sys/fs/file-nr 2>/dev/null)
|
||||
FD_OPEN=$(echo "$FD_LINE" | awk '{print $1}')
|
||||
FD_MAX=$(echo "$FD_LINE" | awk '{print $3}')
|
||||
if [[ -n "$FD_OPEN" && -n "$FD_MAX" && "$FD_MAX" -gt 0 ]]; then
|
||||
FD_PCT=$(( FD_OPEN * 100 / FD_MAX ))
|
||||
if [[ "$FD_PCT" -ge "${SYS_WATCHDOG_FD_CRITICAL_PCT:-95}" ]]; then
|
||||
error "File descriptors ${FD_PCT}% exhausted (${FD_OPEN}/${FD_MAX}) — CRITICAL"
|
||||
CRITICAL_TRIGGERS+=("fd_exhaustion=${FD_PCT}%")
|
||||
else
|
||||
log "File descriptors: ${FD_PCT}% (${FD_OPEN}/${FD_MAX})"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── /boot read-only — state and config writes failing silently ────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_BOOT" == true ]]; then
|
||||
BOOT_TEST="/boot/.watchdog_write_test"
|
||||
if ! touch "$BOOT_TEST" 2>/dev/null; then
|
||||
error "/boot is read-only — config writes failing silently — CRITICAL"
|
||||
CRITICAL_TRIGGERS+=("boot_read_only")
|
||||
else
|
||||
rm -f "$BOOT_TEST" 2>/dev/null
|
||||
log "/boot is writable ✅"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Act on CRITICAL triggers immediately ─────────────────────────────────────────────────
|
||||
if [[ ${#CRITICAL_TRIGGERS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_ERROR CRITICAL — IMMEDIATE REBOOT ━━━"
|
||||
for t in "${CRITICAL_TRIGGERS[@]}"; do
|
||||
error " CRITICAL: $t"
|
||||
done
|
||||
do_reboot "critical" "${CRITICAL_TRIGGERS[@]}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==========================================================================================
|
||||
# ━━━ TIER 3 — STANDARD CHECKS (strike system) ━━━
|
||||
# ==========================================================================================
|
||||
|
||||
# ── rootfs standard ──────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_ROOTFS" == true ]]; then
|
||||
ROOTFS_USED=$(df / --output=pcent 2>/dev/null | tail -1 | tr -d ' %')
|
||||
TRIGGERED=false
|
||||
[[ "$ROOTFS_USED" -ge "$SYS_WATCHDOG_ROOTFS_PCT" ]] && TRIGGERED=true
|
||||
run_strike_check "rootfs" "$TRIGGERED" "rootfs ${ROOTFS_USED}%" && \
|
||||
TRIGGERS+=("rootfs=${ROOTFS_USED}%")
|
||||
fi
|
||||
|
||||
# ── /var/log ─────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_LOG" == true ]]; then
|
||||
LOG_USED=$(df -P /var/log 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%')
|
||||
TRIGGERED=false
|
||||
[[ "${LOG_USED:-0}" -ge "$SYS_WATCHDOG_LOG_PCT" ]] && TRIGGERED=true
|
||||
run_strike_check "log" "$TRIGGERED" "/var/log ${LOG_USED}%" && \
|
||||
TRIGGERS+=("log=${LOG_USED}%")
|
||||
fi
|
||||
|
||||
# ── /tmp ─────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_TMP" == true ]]; then
|
||||
TMP_USED=$(df -P /tmp 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%')
|
||||
if [[ "${TMP_USED:-0}" -ge "${SYS_WATCHDOG_TMP_CRITICAL_PCT:-98}" ]]; then
|
||||
# Try to clear before escalating
|
||||
warn "/tmp ${TMP_USED}% — attempting cleanup..."
|
||||
find /tmp -type f -mmin +60 -not -name "*.lock" -delete 2>/dev/null
|
||||
TMP_USED_AFTER=$(df -P /tmp 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%')
|
||||
if [[ "${TMP_USED_AFTER:-0}" -ge "${SYS_WATCHDOG_TMP_CRITICAL_PCT:-98}" ]]; then
|
||||
error "/tmp still ${TMP_USED_AFTER}% after cleanup — adding to triggers"
|
||||
TRIGGERED=true
|
||||
else
|
||||
warn "/tmp cleared to ${TMP_USED_AFTER}% ✅"
|
||||
TRIGGERED=false
|
||||
fi
|
||||
elif [[ "${TMP_USED:-0}" -ge "${SYS_WATCHDOG_TMP_PCT:-90}" ]]; then
|
||||
TRIGGERED=true
|
||||
else
|
||||
TRIGGERED=false
|
||||
fi
|
||||
run_strike_check "tmp" "$TRIGGERED" "/tmp ${TMP_USED}%" && \
|
||||
TRIGGERS+=("tmp=${TMP_USED}%")
|
||||
fi
|
||||
|
||||
# ── RAM — reboot tier only (warn/shutdown/recover handled by resource_watchdog.sh) ──────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_RAM" == true ]]; then
|
||||
MEM_KB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
|
||||
MEM_GB=$(( MEM_KB / 1024 / 1024 ))
|
||||
|
||||
if [[ "$MEM_GB" -lt "$SYS_WATCHDOG_MEM_GB" ]]; then
|
||||
# Tier 2 — bypass strike system if OOM confirms active crisis
|
||||
if [[ "$SYS_WATCHDOG_CHECK_OOM" == true ]] && \
|
||||
[[ "$OOM_DELTA" -ge "$SYS_WATCHDOG_OOM_LIMIT" ]]; then
|
||||
error "RAM ${MEM_GB}GB + ${OOM_DELTA} OOM kills this run — URGENT bypass"
|
||||
OOM_VICTIMS=$(get_oom_victims)
|
||||
URGENT_TRIGGERS=("urgent_low_ram=${MEM_GB}GB" "oom_kills=${OOM_DELTA}")
|
||||
[[ -n "$OOM_VICTIMS" ]] && URGENT_TRIGGERS+=("oom_victims: $OOM_VICTIMS")
|
||||
do_reboot "urgent" "${URGENT_TRIGGERS[@]}"
|
||||
exit 0
|
||||
fi
|
||||
# Standard strike path
|
||||
run_strike_check "ram" true "RAM ${MEM_GB}GB free" && \
|
||||
TRIGGERS+=("low_ram=${MEM_GB}GB")
|
||||
else
|
||||
reset_strikes "ram"
|
||||
log "RAM ${MEM_GB}GB free ✅"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── ZFS ARC ──────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_ARC" == true ]] && [[ -f /proc/spl/kstat/zfs/arcstats ]]; then
|
||||
ARC_SIZE=$(awk '/^size / {print $3}' /proc/spl/kstat/zfs/arcstats)
|
||||
ARC_MAX=$(awk '/^c_max / {print $3}' /proc/spl/kstat/zfs/arcstats)
|
||||
ARC_PCT=$(( ARC_SIZE * 100 / ARC_MAX ))
|
||||
TRIGGERED=false
|
||||
if [[ "$ARC_PCT" -ge "$SYS_WATCHDOG_ARC_PINNED_PCT" ]]; then
|
||||
sync; echo 3 > /proc/sys/vm/drop_caches; sleep 5
|
||||
ARC_AFTER=$(awk '/^size / {print $3}' /proc/spl/kstat/zfs/arcstats)
|
||||
ARC_AFTER_PCT=$(( ARC_AFTER * 100 / ARC_MAX ))
|
||||
[[ "$ARC_AFTER_PCT" -ge "$SYS_WATCHDOG_ARC_RELEASE_PCT" ]] && TRIGGERED=true
|
||||
fi
|
||||
run_strike_check "arc" "$TRIGGERED" "ZFS ARC pinned ${ARC_PCT}%" && \
|
||||
TRIGGERS+=("arc_pinned=${ARC_PCT}%")
|
||||
fi
|
||||
|
||||
# ── CPU temperature ───────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_CPU_TEMP" == true ]]; then
|
||||
CPU_TEMP=""
|
||||
if command -v sensors >/dev/null 2>&1; then
|
||||
CPU_TEMP=$(sensors 2>/dev/null | \
|
||||
grep -i "Package id 0\|Tctl\|CPU Temp" | \
|
||||
awk '{print $NF}' | tr -d '+°C' | head -1)
|
||||
fi
|
||||
if [[ -n "$CPU_TEMP" ]]; then
|
||||
CPU_TEMP_INT=$(printf "%.0f" "$CPU_TEMP")
|
||||
TRIGGERED=false
|
||||
[[ "$CPU_TEMP_INT" -ge "$SYS_WATCHDOG_CPU_TEMP_MAX" ]] && TRIGGERED=true
|
||||
run_strike_check "cpu_temp" "$TRIGGERED" "CPU temp ${CPU_TEMP_INT}°C" && \
|
||||
TRIGGERS+=("cpu_temp=${CPU_TEMP_INT}C")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Load average ─────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_LOAD" == true ]]; then
|
||||
LOAD=$(awk '{print $1}' /proc/loadavg)
|
||||
LOAD_INT=$(printf "%.0f" "$LOAD")
|
||||
LOAD_THRESHOLD=$(( TOTAL_CORES * SYS_WATCHDOG_LOAD_MULTIPLIER ))
|
||||
TRIGGERED=false
|
||||
[[ "$LOAD_INT" -ge "$LOAD_THRESHOLD" ]] && TRIGGERED=true
|
||||
run_strike_check "load" "$TRIGGERED" "load avg ${LOAD}" && \
|
||||
TRIGGERS+=("load=${LOAD}")
|
||||
fi
|
||||
|
||||
# ── Zombie processes ─────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_ZOMBIES" == true ]]; then
|
||||
ZOMBIE_COUNT=$(ps aux 2>/dev/null | awk '{print $8}' | grep -c "^Z$" || echo 0)
|
||||
ZOMBIE_COUNT="${ZOMBIE_COUNT//[^0-9]/}"; ZOMBIE_COUNT="${ZOMBIE_COUNT:-0}"
|
||||
TRIGGERED=false
|
||||
[[ "$ZOMBIE_COUNT" -ge "$SYS_WATCHDOG_ZOMBIE_LIMIT" ]] && TRIGGERED=true
|
||||
run_strike_check "zombies" "$TRIGGERED" "zombies ${ZOMBIE_COUNT}" && \
|
||||
TRIGGERS+=("zombies=${ZOMBIE_COUNT}")
|
||||
fi
|
||||
|
||||
# ── Array disk errors — accumulating mdstat errors ────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_MDSTAT" == true ]]; then
|
||||
PREV_MD_ERRORS=$(get_state_val "mdstat_errors")
|
||||
CURRENT_MD_ERRORS=$(grep -oP "(?<=\[)[^\]]*[U_][^\]]*(?=\])" \
|
||||
/proc/mdstat 2>/dev/null | grep -o "_" | wc -l || echo 0)
|
||||
CURRENT_MD_ERRORS="${CURRENT_MD_ERRORS//[^0-9]/}"; CURRENT_MD_ERRORS="${CURRENT_MD_ERRORS:-0}"
|
||||
set_state_val "mdstat_errors" "$CURRENT_MD_ERRORS"
|
||||
|
||||
if [[ -n "$PREV_MD_ERRORS" && "$PREV_MD_ERRORS" =~ ^[0-9]+$ ]]; then
|
||||
MD_DELTA=$(( CURRENT_MD_ERRORS - PREV_MD_ERRORS ))
|
||||
if [[ "$MD_DELTA" -ge "${SYS_WATCHDOG_MDSTAT_ERROR_LIMIT:-5}" ]]; then
|
||||
TRIGGERED=true
|
||||
run_strike_check "mdstat" "$TRIGGERED" \
|
||||
"mdstat errors +${MD_DELTA} (total: ${CURRENT_MD_ERRORS})" && \
|
||||
TRIGGERS+=("mdstat_errors=+${MD_DELTA}")
|
||||
else
|
||||
run_strike_check "mdstat" false "mdstat" > /dev/null
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Network interface state ───────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_NETWORK" == true ]]; then
|
||||
NIC="${SYS_WATCHDOG_NIC:-eth0}"
|
||||
NIC_STATE=$(cat "/sys/class/net/${NIC}/operstate" 2>/dev/null || echo "unknown")
|
||||
TRIGGERED=false
|
||||
[[ "$NIC_STATE" != "up" ]] && TRIGGERED=true
|
||||
run_strike_check "network" "$TRIGGERED" "${NIC} state: ${NIC_STATE}" && \
|
||||
TRIGGERS+=("nic_down=${NIC}")
|
||||
fi
|
||||
|
||||
# ── sshd — try restart before escalating ─────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_SSHD" == true ]]; then
|
||||
if ! pgrep -x sshd >/dev/null 2>&1; then
|
||||
warn "sshd not running — attempting restart..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
/etc/rc.d/rc.sshd start >/dev/null 2>&1
|
||||
sleep 3
|
||||
if pgrep -x sshd >/dev/null 2>&1; then
|
||||
warn "sshd restarted successfully ✅"
|
||||
reset_strikes "sshd"
|
||||
notify "sshd was down on $(hostname) ($MY_ID) — restarted automatically" \
|
||||
"System Watchdog" "warning"
|
||||
else
|
||||
error "sshd restart failed — remote access unavailable"
|
||||
run_strike_check "sshd" true "sshd not running" && \
|
||||
TRIGGERS+=("sshd_down")
|
||||
fi
|
||||
else
|
||||
warn "DRY RUN — would restart sshd"
|
||||
fi
|
||||
else
|
||||
reset_strikes "sshd"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Runaway process ───────────────────────────────────────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_RUNAWAY" == true ]]; then
|
||||
RUNAWAY_PCT="${SYS_WATCHDOG_RUNAWAY_CPU_PCT:-90}"
|
||||
TOP_CPU_PCT=$(ps aux 2>/dev/null | awk 'NR>1{print $3}' | sort -rn | head -1)
|
||||
TOP_CPU_INT=$(printf "%.0f" "${TOP_CPU_PCT:-0}")
|
||||
TOP_CPU_NAME=$(ps aux 2>/dev/null | sort -k3 -rn | awk 'NR==2{print $11}')
|
||||
TRIGGERED=false
|
||||
[[ "$TOP_CPU_INT" -ge "$RUNAWAY_PCT" ]] && TRIGGERED=true
|
||||
# Runaway uses SYS_WATCHDOG_RUNAWAY_STRIKES not global strike limit
|
||||
if [[ "$TRIGGERED" == true ]]; then
|
||||
RAWAY_S=$(increment_strikes "runaway")
|
||||
RLIMIT="${SYS_WATCHDOG_RUNAWAY_STRIKES:-3}"
|
||||
warn "Runaway ${TOP_CPU_NAME} ${TOP_CPU_PCT}% CPU -- strike $RAWAY_S/$RLIMIT"
|
||||
if (( RAWAY_S >= RLIMIT )); then
|
||||
error "Runaway process ${TOP_CPU_NAME} -- strike limit hit"
|
||||
reset_strikes "runaway"
|
||||
TRIGGERS+=("runaway=${TOP_CPU_NAME}@${TOP_CPU_PCT}%")
|
||||
fi
|
||||
else
|
||||
RAWAY_CUR=$(get_strikes "runaway")
|
||||
[[ "${RAWAY_CUR:-0}" -gt 0 ]] && reset_strikes "runaway"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Required containers from docker_watchdog skip list ────────────────────────────────────
|
||||
if [[ "$SYS_WATCHDOG_CHECK_CONTAINERS" == true ]] && [[ -s "$SYS_WATCHDOG_FAILED_FILE" ]]; then
|
||||
FAILED_CONTAINERS=()
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
|
||||
'{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
|
||||
[[ "$STATUS" != "true" ]] && FAILED_CONTAINERS+=("$container")
|
||||
done < "$SYS_WATCHDOG_FAILED_FILE"
|
||||
|
||||
TRIGGERED=false
|
||||
[[ ${#FAILED_CONTAINERS[@]} -gt 0 ]] && TRIGGERED=true
|
||||
run_strike_check "failed_containers" "$TRIGGERED" \
|
||||
"required containers stopped: ${FAILED_CONTAINERS[*]:-}" && \
|
||||
TRIGGERS+=("containers=${FAILED_CONTAINERS[*]:-}")
|
||||
fi
|
||||
|
||||
# ==========================================================================================
|
||||
# ━━━ Evaluate Standard Triggers ━━━
|
||||
# ==========================================================================================
|
||||
if [[ ${#TRIGGERS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_REBOOT_SMART System Watchdog — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
for t in "${TRIGGERS[@]}"; do
|
||||
echo " $ICON_REBOOT_SMART $t"
|
||||
done
|
||||
[[ "$OOM_DELTA" -gt 0 ]] && echo " OOM kills this run: $OOM_DELTA"
|
||||
echo ""
|
||||
do_reboot "standard" "${TRIGGERS[@]}"
|
||||
exit 0
|
||||
else
|
||||
echo "System healthy ✅ ($(date '+%H:%M:%S'))"
|
||||
fi
|
||||
|
||||
# Keep state file mtime fresh — docker_watchdog stale guard checks this
|
||||
set_state_val "watchdog_cycle" "$(date +%s)"
|
||||
Executable
+194
@@ -0,0 +1,194 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================ Watchdog Orchestrator ===========================================
|
||||
# ==============================================================================================
|
||||
# Runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
|
||||
# Schedule: * * * * * (every minute via User Scripts plugin)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# Driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf — add, remove, or reorder there.
|
||||
# Default: resource_watchdog → docker_watchdog → system_watchdog
|
||||
#
|
||||
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
|
||||
# Resource Watchdog first — frees RAM and CPU before healing attempts container restarts.
|
||||
# Containers restarted into a resource-pressured system just fail again.
|
||||
# Docker Watchdog second — restarts with pressure already reduced, more likely to stabilise.
|
||||
# System Watchdog last — only triggers if prior layers could not resolve the issue.
|
||||
# Rebooting without first reducing pressure may reboot into the same state.
|
||||
#
|
||||
# ── STARTUP GRACE ─────────────────────────────────────────────────────────────────────────────
|
||||
# No action until system uptime >= WATCHDOG_STARTUP_GRACE seconds.
|
||||
# Prevents false positives from containers still starting at array launch.
|
||||
# Each sub-script enforces this independently — orchestrator exits early to avoid log noise.
|
||||
#
|
||||
# ── OVERLAP PROTECTION ────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock() — exits immediately if a prior cycle is still in progress.
|
||||
# Prevents pile-up when a cycle runs long (daemon restart attempt = 30s, etc.).
|
||||
#
|
||||
# ── REPLACES ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Continuous loops previously in system_watchdog.sh and docker_watchdog.sh.
|
||||
# Those scripts are now single-pass — this orchestrator provides the cadence.
|
||||
# Remove system_watchdog.sh and docker_watchdog.sh from ARRAY_START_SCRIPTS.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# WATCHDOG_ORCHESTRATOR_SCRIPTS — watchdogs to run, in order
|
||||
# WATCHDOG_STARTUP_GRACE — seconds after boot before checks activate
|
||||
# WATCHDOG_ORCHESTRATOR_HEARTBEAT — periodic heartbeat log toggle
|
||||
# WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS — heartbeat interval in hours
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# watchdog_orchestrator.sh — normal run (called by cron every minute)
|
||||
# watchdog_orchestrator.sh --dry-run — pass --dry-run to all sub-scripts
|
||||
# watchdog_orchestrator.sh --status — show script paths and current grace state
|
||||
# watchdog_orchestrator.sh --log — verbose output from all sub-scripts
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
|
||||
source "$ECOSYSTEM_ROOT/load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Skip immediately if another cycle is still running — no pile-up
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing --dry-run to all sub-scripts"
|
||||
|
||||
# Derive a display name from a script path: "resource_watchdog.sh" → "Resource Watchdog"
|
||||
_watchdog_display_name() {
|
||||
local path="$1"
|
||||
local base="${path##*/}"
|
||||
base="${base%.sh}"
|
||||
base="${base//_/ }"
|
||||
echo "$base" | awk '{for(i=1;i<=NF;i++) $i=toupper(substr($i,1,1)) substr($i,2); print}'
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY WATCHDOG ORCHESTRATOR STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
|
||||
UPTIME_S=$(awk '{print int($1)}' /proc/uptime)
|
||||
if [[ "$UPTIME_S" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
|
||||
warn "Within startup grace — $(format_duration $UPTIME_S) / $(format_duration $WATCHDOG_STARTUP_GRACE)"
|
||||
else
|
||||
echo "Past startup grace — $(format_duration $UPTIME_S) uptime"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "── Sub-scripts ──"
|
||||
for entry in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do
|
||||
local_path="$ECOSYSTEM_ROOT/$entry"
|
||||
label="$(_watchdog_display_name "$entry")"
|
||||
if [[ -f "$local_path" ]]; then
|
||||
[[ -x "$local_path" ]] && icon="$ICON_DONE" || icon="$ICON_WARN"
|
||||
echo " $icon $label — ${local_path##*/}"
|
||||
else
|
||||
echo " $ICON_ERROR $label — NOT FOUND: $local_path"
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo " Schedule: * * * * * (every minute via User Scripts)"
|
||||
echo " Heartbeat: ${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true} / every ${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Startup Grace ━━━
|
||||
# ==============================================================================================
|
||||
UPTIME_SECONDS=$(awk '{print int($1)}' /proc/uptime)
|
||||
if [[ "$UPTIME_SECONDS" -lt "$WATCHDOG_STARTUP_GRACE" ]]; then
|
||||
log "Startup grace — ${UPTIME_SECONDS}s / ${WATCHDOG_STARTUP_GRACE}s — skipping cycle"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Run Watchdog Cycle ━━━
|
||||
# ==============================================================================================
|
||||
CYCLE_START=$(date +%s)
|
||||
PASS=()
|
||||
FAIL=()
|
||||
|
||||
run_watchdog() {
|
||||
local name="$1" script="$2"
|
||||
|
||||
if [[ ! -f "$script" ]]; then
|
||||
error "$name — not found: $script"
|
||||
FAIL+=("$name:missing")
|
||||
return 1
|
||||
fi
|
||||
|
||||
[[ ! -x "$script" ]] && chmod +x "$script"
|
||||
|
||||
local extra_args=()
|
||||
[[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
|
||||
[[ "$VERBOSE" == true ]] && extra_args+=("--log")
|
||||
|
||||
log "$ICON_START $name"
|
||||
if bash "$script" "${extra_args[@]}"; then
|
||||
PASS+=("$name")
|
||||
return 0
|
||||
else
|
||||
error "$name — non-zero exit"
|
||||
FAIL+=("$name")
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
for _entry in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do
|
||||
run_watchdog "$(_watchdog_display_name "$_entry")" "$ECOSYSTEM_ROOT/$_entry"
|
||||
done
|
||||
|
||||
CYCLE_END=$(date +%s)
|
||||
DURATION=$(( CYCLE_END - CYCLE_START ))
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Heartbeat ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}" == true ]]; then
|
||||
HB_SECONDS=$(( ${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1} * 3600 ))
|
||||
HB_COUNT_FILE="/tmp/watchdog_orch_hb.count"
|
||||
HB_COUNT=$(cat "$HB_COUNT_FILE" 2>/dev/null || echo 0)
|
||||
HB_COUNT=$(( HB_COUNT + 1 ))
|
||||
echo "$HB_COUNT" > "$HB_COUNT_FILE"
|
||||
# Each cron run = ~60s — use count × 60 as uptime approximation
|
||||
HB_ELAPSED=$(( HB_COUNT * 60 ))
|
||||
if [[ "$HB_SECONDS" -gt 0 ]] && (( HB_ELAPSED % HB_SECONDS < 60 )) && [[ "$HB_COUNT" -gt 1 ]]; then
|
||||
HB_HR=$(( HB_ELAPSED / 3600 ))
|
||||
warn "♥ watchdog_orchestrator alive — $MY_ID — ~${HB_HR}hr ($(date '+%H:%M:%S'))"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary — only shown on failures or --log ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "${#FAIL[@]}" -gt 0 || "$VERBOSE" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY WATCHDOG CYCLE — $MY_ID — $(date '+%H:%M:%S') ━━━━━"
|
||||
for p in "${PASS[@]}"; do log " $ICON_DONE $p"; done
|
||||
for f in "${FAIL[@]}"; do error " $ICON_ERROR $f"; done
|
||||
echo "$ICON_TIME Duration: $(format_duration $DURATION)"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
if [[ "${#FAIL[@]}" -gt 0 ]]; then
|
||||
notify "Watchdog cycle failure on $(hostname) ($MY_ID) — ${FAIL[*]}" \
|
||||
"Watchdog Orchestrator" "warning"
|
||||
fi
|
||||
fi
|
||||
@@ -0,0 +1,286 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# =========================== Watchdog Skip List Manager =======================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# View and manage the persistent container skip list used by docker_watchdog.sh.
|
||||
# docker_watchdog.sh adds a container to the skip list when it exceeds
|
||||
# WATCHDOG_CONTAINER_RESTART_LIMIT restarts within WATCHDOG_CONTAINER_RESTART_WINDOW
|
||||
# hours — prevents infinite restart loops on containers that keep crashing.
|
||||
#
|
||||
# Skip list persists on /boot/config (survives reboots). Auto-clears when
|
||||
# docker_watchdog.sh sees the container running on a later cycle. Use this
|
||||
# script to clear manually after fixing the underlying problem.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Skip List Lifecycle
|
||||
# 1. Container crashes repeatedly → watchdog adds to skip list, notifies
|
||||
# 2. Watchdog stops restarting the container on subsequent cycles
|
||||
# 3a. If container recovers on its own (Docker restart policy), watchdog
|
||||
# sees it running, removes from skip list automatically
|
||||
# 3b. If stuck stopped → fix the root cause, clear via this script, then
|
||||
# docker start ContainerName manually
|
||||
# 4. Watchdog monitors normally on next cycle. If it crashes again → re-added.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent access with docker_watchdog.sh writing
|
||||
# the same files.
|
||||
#
|
||||
# Active Watchdog Detection
|
||||
# Warns if docker_watchdog.sh is currently running when a clear is attempted
|
||||
# — the watchdog could re-add the container to the skip list within seconds.
|
||||
#
|
||||
# Docker Timeout
|
||||
# DOCKER_TIMEOUT caps docker inspect calls against a hung daemon.
|
||||
#
|
||||
# Confirmation Required
|
||||
# Interactive mode prompts for YES before clearing. Use --force for scripts.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# SYS_WATCHDOG_FAILED_FILE — persistent container skip list (on /boot/config)
|
||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history used for loop detection
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# watchdog_skip_list_manager.sh [--status]
|
||||
# Show skip list, container states, and recent restart history.
|
||||
#
|
||||
# watchdog_skip_list_manager.sh --clear ContainerName
|
||||
# Remove a specific container from the skip list and clear its restart history.
|
||||
# Prompts for YES unless --force is passed.
|
||||
#
|
||||
# watchdog_skip_list_manager.sh --clear-all
|
||||
# Clear all skip lists and all restart history.
|
||||
# Prompts for YES unless --force is passed.
|
||||
#
|
||||
# All actions support --dry-run (show what would change) and --force (skip prompt).
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
DOCKER_TIMEOUT=15
|
||||
|
||||
# ── Parse action flags before parse_args ──────────────────────────────────────────────────────
|
||||
ACTION="status"
|
||||
TARGET_CONTAINER=""
|
||||
FORCE=false
|
||||
FILTERED_ARGS=()
|
||||
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--clear-all) ACTION="clear-all" ;;
|
||||
--clear) ACTION="clear" ;;
|
||||
--status) ACTION="status" ;;
|
||||
--force) FORCE=true ;;
|
||||
*)
|
||||
if [[ "$ACTION" == "clear" && -z "$TARGET_CONTAINER" ]]; then
|
||||
TARGET_CONTAINER="$arg"
|
||||
else
|
||||
FILTERED_ARGS+=("$arg")
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${FILTERED_ARGS[@]}"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
validate_unraid_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# detect_hosts() sets MY_ID — used in output
|
||||
detect_hosts
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
[[ "$FORCE" == true ]] && warn "FORCE mode — confirmation prompt skipped"
|
||||
|
||||
# Ensure state files exist
|
||||
touch "$SYS_WATCHDOG_FAILED_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status — always shown regardless of action ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_WATCHDOG Skip List Status — $MY_ID ━━━"
|
||||
|
||||
SKIP_COUNT=$(grep -c "." "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null || echo 0)
|
||||
SKIP_COUNT="${SKIP_COUNT//[^0-9]/}"; SKIP_COUNT="${SKIP_COUNT:-0}"
|
||||
RESTART_COUNT=$(wc -l < "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null || echo 0)
|
||||
RESTART_COUNT="${RESTART_COUNT//[^0-9]/}"; RESTART_COUNT="${RESTART_COUNT:-0}"
|
||||
|
||||
# docker_watchdog.sh running check
|
||||
WATCHDOG_RUNNING=false
|
||||
if pgrep -f "docker_watchdog.sh" >/dev/null 2>&1; then
|
||||
WATCHDOG_RUNNING=true
|
||||
warn "docker_watchdog.sh is currently RUNNING"
|
||||
[[ "$ACTION" != "status" ]] && \
|
||||
warn "Clearing during an active cycle — watchdog may re-add container on next iteration"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
if [[ "$SKIP_COUNT" -eq 0 ]]; then
|
||||
echo "Skip list: empty — all containers monitored normally ✅"
|
||||
else
|
||||
warn "$SKIP_COUNT container(s) on skip list — manual intervention needed:"
|
||||
echo ""
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
|
||||
'{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
|
||||
case "$STATUS" in
|
||||
true)
|
||||
echo " $ICON_RUNNING $container — RUNNING (watchdog will auto-clear next cycle)"
|
||||
;;
|
||||
false)
|
||||
echo " $ICON_NOT_RUNNING $container — STOPPED — fix and start manually"
|
||||
;;
|
||||
*)
|
||||
echo " $ICON_WARN $container — not found on this server"
|
||||
;;
|
||||
esac
|
||||
done < "$SYS_WATCHDOG_FAILED_FILE"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_WATCHDOG Restart History ━━━"
|
||||
if [[ "$RESTART_COUNT" -eq 0 ]]; then
|
||||
echo "No restart history"
|
||||
else
|
||||
echo "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)"
|
||||
echo ""
|
||||
awk -F'|' '{counts[$1]++} END {
|
||||
for (c in counts)
|
||||
printf " %-30s %d restart(s)\n", c, counts[c]
|
||||
}' "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null | sort
|
||||
fi
|
||||
|
||||
[[ "$ACTION" == "status" ]] && exit 0
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Clear All ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$ACTION" == "clear-all" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_TRASH Clear All Skip Lists ━━━"
|
||||
warn "This will clear the skip list and restart history for ALL containers"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if [[ "$FORCE" == true ]]; then
|
||||
log "FORCE flag set — skipping confirmation"
|
||||
elif [[ -t 0 ]]; then
|
||||
read -r -p "Type YES to confirm: " CONFIRM
|
||||
if [[ "$CONFIRM" != "YES" ]]; then
|
||||
warn "Cancelled"
|
||||
exit 0
|
||||
fi
|
||||
else
|
||||
error "Non-interactive mode — use --force flag to skip confirmation"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
> "$SYS_WATCHDOG_FAILED_FILE"
|
||||
> "$WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
warn "Skip list cleared ✅"
|
||||
warn "Restart history cleared ✅"
|
||||
[[ "$WATCHDOG_RUNNING" == true ]] && \
|
||||
warn "Note: watchdog is running — containers will be monitored on next cycle"
|
||||
notify "Watchdog skip list cleared on $(hostname) ($MY_ID) — all containers will be monitored normally" \
|
||||
"Watchdog Manager" "warning"
|
||||
else
|
||||
warn "DRY RUN — would clear: $SYS_WATCHDOG_FAILED_FILE"
|
||||
warn "DRY RUN — would clear: $WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Clear Specific Container ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$ACTION" == "clear" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_TRASH Clear Container: $TARGET_CONTAINER ━━━"
|
||||
|
||||
if [[ -z "$TARGET_CONTAINER" ]]; then
|
||||
error "No container specified"
|
||||
error "Usage: watchdog_skip_list_manager.sh --clear ContainerName"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Remove from skip list
|
||||
if ! grep -q "^${TARGET_CONTAINER}$" "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null; then
|
||||
warn "$TARGET_CONTAINER is not on the skip list"
|
||||
else
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
sed -i "/^${TARGET_CONTAINER}$/d" "$SYS_WATCHDOG_FAILED_FILE"
|
||||
warn "$TARGET_CONTAINER removed from skip list ✅"
|
||||
else
|
||||
warn "DRY RUN — would remove $TARGET_CONTAINER from skip list"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Clear restart history for this container
|
||||
HIST_COUNT=$(grep -c "^${TARGET_CONTAINER}|" \
|
||||
"$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null || echo 0)
|
||||
HIST_COUNT="${HIST_COUNT//[^0-9]/}"; HIST_COUNT="${HIST_COUNT:-0}"
|
||||
|
||||
if [[ "$HIST_COUNT" -gt 0 ]]; then
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
sed -i "/^${TARGET_CONTAINER}|/d" "$WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
warn "Cleared $HIST_COUNT restart history entries for $TARGET_CONTAINER ✅"
|
||||
else
|
||||
warn "DRY RUN — would clear $HIST_COUNT restart history entries"
|
||||
fi
|
||||
else
|
||||
log "No restart history for $TARGET_CONTAINER"
|
||||
fi
|
||||
|
||||
[[ "$WATCHDOG_RUNNING" == true ]] && \
|
||||
warn "Note: watchdog is running — $TARGET_CONTAINER may be re-added if still failing"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_INFO Next Steps ━━━"
|
||||
echo " 1. Fix whatever was causing $TARGET_CONTAINER to fail"
|
||||
echo " 2. Start it manually: docker start $TARGET_CONTAINER"
|
||||
echo " 3. docker_watchdog.sh monitors it on the next cycle"
|
||||
echo " 4. If it crashes again → watchdog adds it back and notifies"
|
||||
notify "$TARGET_CONTAINER cleared from watchdog skip list on $(hostname) ($MY_ID)" \
|
||||
"Watchdog Manager" "warning"
|
||||
fi
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DONE — $MY_ID ━━━━━"
|
||||
Reference in New Issue
Block a user