feat: slskd reconnect guard in downloaders_reset, mass v2 sync

- downloaders_reset: connection check block before slskd API sections;
  triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
  gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
  Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
  common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
2026-05-19 20:00:10 -04:00
parent 5cb16d4b18
commit e13f2fa14f
81 changed files with 12164 additions and 10656 deletions
+90 -36
View File
@@ -1,49 +1,103 @@
#!/bin/bash
# ==============================================================================================
# ================================= Backup Verify ==============================================
# ============================= Backup Verify ==================================================
# ==============================================================================================
# Verifies the rsync mirror is healthy by comparing random file samples between
# local and remote servers using MD5 checksums.
#
# ── WHAT IT DOES ──────────────────────────────────────────────────────────────────────────────
# Randomly samples BACKUP_VERIFY_SAMPLE files per share above BACKUP_VERIFY_MIN_SIZE,
# computes MD5 checksums locally, then computes the same checksums on the remote via SSH
# and compares results. Catches silent corruption or incomplete syncs that rsync itself
# would not detect.
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# rsync mirror integrity verification via independent MD5 checksums. Scheduled
# weekly (Sunday 10am). Randomly samples BACKUP_VERIFY_SAMPLE files per share
# above BACKUP_VERIFY_MIN_SIZE, computes checksums locally, then computes the
# same checksums on the remote via SSH and compares.
#
# ── RESULTS PER FILE ──────────────────────────────────────────────────────────────────────────
# MATCH — checksums identical, file is correctly mirrored ✅
# MISMATCH — file exists on both but checksums differ — sync may have partially failed
# MISSING — file exists locally but not on remote — not yet synced or deleted on remote
# Per file: MATCH (checksums identical) | MISMATCH (file exists on both but
# checksums differ — sync failure or corruption) | MISSING (file exists locally
# but not on remote). All MISMATCHes and significant MISSINGs trigger notification.
# rsync exit code 0 is not trustedthis script verifies actual content.
#
# ── SHARE SELECTION ───────────────────────────────────────────────────────────────────────────
# Uses HOST*_BACKUP_VERIFY_SHARES if defined, falls back to HOST*_DAILY_SYNC_SHARES.
# Both aliased by detect_hosts() — no manual HOST1/HOST2 selection needed.
# Share list from HOST*_BACKUP_VERIFY_SHARES if defined, otherwise falls back
# to HOST*_DAILY_SYNC_SHARES. Both aliased by detect_hosts().
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# acquire_lock — prevents concurrent runs producing duplicate/conflicting results
# check_connectivity() — verifies remote reachable before attempting SSH calls
# check_remote_array() — verifies remote array mounted before checksums
# remote array down = all files "missing" = false alarm ✅
# version parity — verifies both servers on compatible unRAID before trusting results
# SSH_TIMEOUT — all SSH calls protected against hangs
# validate_unraid_cmd — notify script validated before use
# Silent by default — only issues produce output, all-match runs are silent
# ==============================================================================================
# DESIGN PRINCIPLES
# ==============================================================================================
#
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
# HOST*_BACKUP_VERIFY_SHARES — override share list (empty = use DAILY_SYNC_SHARES)
# HOST*_DAILY_SYNC_SHARES — fallback share list
# All aliased by detect_hosts()
# Independent Verification
# rsync reports success when the transfer completed without network errors and
# file sizes and modification times match. It does not detect silent corruption
# during transfer (bitflip in transit), corruption written to storage at rest
# (faulty drive sector), or files that matched size/mtime but had wrong content.
# All of these produce exit code 0. This script checks whether "done" means "correct."
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# BACKUP_VERIFY_SAMPLE — random files to check per share (default 10)
# BACKUP_VERIFY_MIN_SIZE — minimum file size to include in sample (default 1M)
# Intentionally Small Sample
# 10 files per share (default) — a spot check, not an exhaustive verify.
# Catches systematic problems and hardware issues while running in minutes, not
# hours. Full verification would take longer than the rsync itself.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Single Instance Lock
# acquire_lock prevents concurrent runs producing conflicting results.
#
# Remote Connectivity Check
# check_connectivity() verifies the remote Tailscale IP is reachable before
# any SSH calls. Without this, all files show as MISSING on a network hiccup.
#
# Remote Array Check
# check_remote_array() verifies /mnt/user is mounted on the remote before
# computing checksums. Array not started = all files "missing" = false alarm.
#
# Version Parity
# Refuses to run if remote unRAID version doesn't match local. A mismatch
# may mean the remote is in an unexpected state.
#
# SSH Timeout
# SSH_TIMEOUT caps all SSH calls. One hung connection does not block the run.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================
#
# master_host*.conf
#
# HOST*_BACKUP_VERIFY_SHARES
# Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
# Aliased by detect_hosts() → BACKUP_VERIFY_SHARES.
#
# HOST*_DAILY_SYNC_SHARES
# Fallback share list if BACKUP_VERIFY_SHARES is empty. Aliased by detect_hosts().
#
# master.conf
#
# BACKUP_VERIFY_SAMPLE
# Random files checked per share per run. (default: 10)
#
# BACKUP_VERIFY_MIN_SIZE
# Minimum file size to include in sample — tiny files have low corruption
# risk and slow checksums. (default: 1M)
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# backup_verify.sh
# Sample files from all shares and compare checksums. Notify on MISMATCH
# or significant MISSING count. Silent when all samples match.
#
# backup_verify.sh --dry-run
# Show which files would be sampled. No checksums computed, no notifications.
#
# backup_verify.sh --status
# Show share list, sample size, and min file size configuration. Then exit.
#
# backup_verify.sh --log
# Verbose per-file checksum comparison output during the run.
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# backup_verify.sh — normal run
# backup_verify.sh --dry-run — show sample selection only, no checksums
# backup_verify.sh --log — verbose output
# backup_verify.sh --status — show config and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"