feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
+90
-36
@@ -1,49 +1,103 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Backup Verify ==============================================
|
||||
# ============================= Backup Verify ==================================================
|
||||
# ==============================================================================================
|
||||
# Verifies the rsync mirror is healthy by comparing random file samples between
|
||||
# local and remote servers using MD5 checksums.
|
||||
#
|
||||
# ── WHAT IT DOES ──────────────────────────────────────────────────────────────────────────────
|
||||
# Randomly samples BACKUP_VERIFY_SAMPLE files per share above BACKUP_VERIFY_MIN_SIZE,
|
||||
# computes MD5 checksums locally, then computes the same checksums on the remote via SSH
|
||||
# and compares results. Catches silent corruption or incomplete syncs that rsync itself
|
||||
# would not detect.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# rsync mirror integrity verification via independent MD5 checksums. Scheduled
|
||||
# weekly (Sunday 10am). Randomly samples BACKUP_VERIFY_SAMPLE files per share
|
||||
# above BACKUP_VERIFY_MIN_SIZE, computes checksums locally, then computes the
|
||||
# same checksums on the remote via SSH and compares.
|
||||
#
|
||||
# ── RESULTS PER FILE ──────────────────────────────────────────────────────────────────────────
|
||||
# MATCH — checksums identical, file is correctly mirrored ✅
|
||||
# MISMATCH — file exists on both but checksums differ — sync may have partially failed
|
||||
# MISSING — file exists locally but not on remote — not yet synced or deleted on remote
|
||||
# Per file: MATCH (checksums identical) | MISMATCH (file exists on both but
|
||||
# checksums differ — sync failure or corruption) | MISSING (file exists locally
|
||||
# but not on remote). All MISMATCHes and significant MISSINGs trigger notification.
|
||||
# rsync exit code 0 is not trusted — this script verifies actual content.
|
||||
#
|
||||
# ── SHARE SELECTION ───────────────────────────────────────────────────────────────────────────
|
||||
# Uses HOST*_BACKUP_VERIFY_SHARES if defined, falls back to HOST*_DAILY_SYNC_SHARES.
|
||||
# Both aliased by detect_hosts() — no manual HOST1/HOST2 selection needed.
|
||||
# Share list from HOST*_BACKUP_VERIFY_SHARES if defined, otherwise falls back
|
||||
# to HOST*_DAILY_SYNC_SHARES. Both aliased by detect_hosts().
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent runs producing duplicate/conflicting results
|
||||
# check_connectivity() — verifies remote reachable before attempting SSH calls
|
||||
# check_remote_array() — verifies remote array mounted before checksums
|
||||
# remote array down = all files "missing" = false alarm ✅
|
||||
# version parity — verifies both servers on compatible unRAID before trusting results
|
||||
# SSH_TIMEOUT — all SSH calls protected against hangs
|
||||
# validate_unraid_cmd — notify script validated before use
|
||||
# Silent by default — only issues produce output, all-match runs are silent
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_BACKUP_VERIFY_SHARES — override share list (empty = use DAILY_SYNC_SHARES)
|
||||
# HOST*_DAILY_SYNC_SHARES — fallback share list
|
||||
# All aliased by detect_hosts()
|
||||
# Independent Verification
|
||||
# rsync reports success when the transfer completed without network errors and
|
||||
# file sizes and modification times match. It does not detect silent corruption
|
||||
# during transfer (bitflip in transit), corruption written to storage at rest
|
||||
# (faulty drive sector), or files that matched size/mtime but had wrong content.
|
||||
# All of these produce exit code 0. This script checks whether "done" means "correct."
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# BACKUP_VERIFY_SAMPLE — random files to check per share (default 10)
|
||||
# BACKUP_VERIFY_MIN_SIZE — minimum file size to include in sample (default 1M)
|
||||
# Intentionally Small Sample
|
||||
# 10 files per share (default) — a spot check, not an exhaustive verify.
|
||||
# Catches systematic problems and hardware issues while running in minutes, not
|
||||
# hours. Full verification would take longer than the rsync itself.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs producing conflicting results.
|
||||
#
|
||||
# Remote Connectivity Check
|
||||
# check_connectivity() verifies the remote Tailscale IP is reachable before
|
||||
# any SSH calls. Without this, all files show as MISSING on a network hiccup.
|
||||
#
|
||||
# Remote Array Check
|
||||
# check_remote_array() verifies /mnt/user is mounted on the remote before
|
||||
# computing checksums. Array not started = all files "missing" = false alarm.
|
||||
#
|
||||
# Version Parity
|
||||
# Refuses to run if remote unRAID version doesn't match local. A mismatch
|
||||
# may mean the remote is in an unexpected state.
|
||||
#
|
||||
# SSH Timeout
|
||||
# SSH_TIMEOUT caps all SSH calls. One hung connection does not block the run.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_BACKUP_VERIFY_SHARES
|
||||
# Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
|
||||
# Aliased by detect_hosts() → BACKUP_VERIFY_SHARES.
|
||||
#
|
||||
# HOST*_DAILY_SYNC_SHARES
|
||||
# Fallback share list if BACKUP_VERIFY_SHARES is empty. Aliased by detect_hosts().
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# BACKUP_VERIFY_SAMPLE
|
||||
# Random files checked per share per run. (default: 10)
|
||||
#
|
||||
# BACKUP_VERIFY_MIN_SIZE
|
||||
# Minimum file size to include in sample — tiny files have low corruption
|
||||
# risk and slow checksums. (default: 1M)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# backup_verify.sh
|
||||
# Sample files from all shares and compare checksums. Notify on MISMATCH
|
||||
# or significant MISSING count. Silent when all samples match.
|
||||
#
|
||||
# backup_verify.sh --dry-run
|
||||
# Show which files would be sampled. No checksums computed, no notifications.
|
||||
#
|
||||
# backup_verify.sh --status
|
||||
# Show share list, sample size, and min file size configuration. Then exit.
|
||||
#
|
||||
# backup_verify.sh --log
|
||||
# Verbose per-file checksum comparison output during the run.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# backup_verify.sh — normal run
|
||||
# backup_verify.sh --dry-run — show sample selection only, no checksums
|
||||
# backup_verify.sh --log — verbose output
|
||||
# backup_verify.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user