feat: slskd reconnect guard in downloaders_reset, mass v2 sync

- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
2026-05-19 20:00:10 -04:00
parent 5cb16d4b18
commit e13f2fa14f
81 changed files with 12164 additions and 10656 deletions
@@ -1,54 +1,101 @@
 #!/bin/bash
 # ==============================================================================================
-# ================================= Failover Test ==============================================
+# ================================= Fallback Test ==============================================
 # ==============================================================================================
-# Controlled simulation of the failover lifecycle — validates the entire failover sequence
-# without waiting for a real outage.
 #
-# ── WHAT THIS SCRIPT IS ───────────────────────────────────────────────────────────────────────
-# A test harness only — contains no failover logic.
-# All failover logic lives in fallback.sh and is exercised by this test.
-# Any changes to fallback.sh are automatically reflected here.
+# PURPOSE
+# ─────────────────────────────────────────────────────────────────────────────
+# Controlled simulation of the fallback lifecycle. Validates the entire sequence
+# without waiting for a real outage. Contains no fallback logic — exercises the
+# real fallback.sh via an iptables DROP rule on the remote Tailscale IP.
 #
-# ── TEST SEQUENCE ─────────────────────────────────────────────────────────────────────────────
-#   Phase 1 — Pre-flight         verify both servers reachable, daemons healthy,
-#                                 version parity, fallback.sh exists, state is NORMAL
-#   Phase 2 — Block Remote       iptables rule drops all traffic to remote IP
-#   Phase 3 — Fallback Detection wait for fallback.sh to detect outage and enter FALLBACK
-#   Phase 4 — Container Start    verify Tier 1 failover containers started locally
-#   Phase 5 — Restore            remove iptables rule, remote becomes reachable
-#   Phase 6 — Handback           wait for fallback.sh to complete handback to NORMAL
-#   Phase 7 — Container Handback verify Tier 1 containers stopped locally after handback
-#   Phase 8 — Report             full pass/fail summary per phase
+# Run during a maintenance window. Users will experience a brief service
+# interruption. Use --dry-run to walk through all phases without real changes.
 #
-# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
-#   FALLBACK_ENABLED gate   — aborts if fallback monitoring is disabled
-#   iptables safety trap    — rule ALWAYS removed on exit (crash, error, ctrl-c, normal)
-#                             remote connectivity always restored regardless of outcome
-#   Version parity check    — pre-flight verifies both servers on compatible unRAID versions
-#   Remote Docker daemon    — pre-flight verifies remote daemon is responsive
-#   DOCKER_TIMEOUT          — all docker calls protected against daemon hangs
-#   MY_ID-based routing     — tier containers selected via MY_ID not hostname comparison
-#   Command validation      — iptables and notify validated before use
-#   Dry-run safe            — full sequence walkthrough without touching iptables or containers
+# ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
 #
-# ── WARNING ───────────────────────────────────────────────────────────────────────────────────
-# ⚠️  This script starts and stops REAL containers on both servers.
-#     Run during a maintenance window — users will experience a brief service interruption.
-#     Use --dry-run to walk through the sequence without any real changes.
+# Phase 1 — Pre-flight         Both servers reachable, Docker daemons healthy,
+#                               version parity, fallback.sh exists, state NORMAL
+# Phase 2 — Block Remote        iptables DROP rule added — remote appears unreachable
+# Phase 3 — Fallback Detection  Wait FALLBACK_TEST_BLOCK_WAIT for fallback.sh to
+#                               detect the outage and enter FALLBACK state
+# Phase 4 — Container Start     Verify Tier 1 containers started locally
+# Phase 5 — Restore             iptables rule removed — remote reachable again
+# Phase 6 — Handback            Wait FALLBACK_TEST_HANDBACK_WAIT for fallback.sh to
+#                               complete full handback and return to NORMAL
+# Phase 7 — Container Handback  Verify Tier 1 containers stopped locally
+# Report  — Full pass/fail per phase with timing
 #
-# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
-#   FALLBACK_TEST_BLOCK_WAIT    — seconds to wait for fallback.sh to detect outage
-#   FALLBACK_TEST_HANDBACK_WAIT — seconds to wait for fallback.sh to complete handback
-#   FALLBACK_CHECK_INTERVAL     — check interval of the running fallback.sh (informational)
-#   FALLBACK_HANDBACK_STRIKES   — strikes required before handback (informational)
-#   FALLBACK_STATE_FILE         — state file path to read current state
+# ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Test Harness Only
+#   Contains zero fallback logic. All fallback is exercised through fallback.sh.
+#   Any change to fallback.sh is automatically reflected in the test result.
+#
+# ==============================================================================================
+# OPERATIONAL SAFEGUARDS
+# ==============================================================================================
+#
+# iptables Safety Trap
+#   The DROP rule is removed via trap on ANY exit — normal completion, crash, error,
+#   ctrl-c. Remote connectivity is always restored regardless of test outcome.
+#   You cannot accidentally leave the remote permanently blocked.
+#
+# FALLBACK_ENABLED Gate
+#   Aborts if FALLBACK_ENABLED=false. Testing a disabled fallback system is
+#   misleading and potentially destructive.
+#
+# State Must Be NORMAL
+#   Pre-flight fails if state is not NORMAL. Running a test during an actual
+#   fallback event would interfere with the real event.
+#
+# Version Parity Check
+#   Pre-flight verifies unRAID version parity before any iptables rules are
+#   added. A mismatch makes the test result unreliable.
+#
+# Remote Docker Daemon Check
+#   Pre-flight confirms remote Docker daemon is responsive before Phase 2.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   FALLBACK_TEST_BLOCK_WAIT
+#       Seconds to wait in Phase 3 for fallback.sh to detect the outage.
+#       Must be > FALLBACK_CHECK_INTERVAL + buffer. At 30s interval: use ≥60s.
+#       (default: 60)
+#
+#   FALLBACK_TEST_HANDBACK_WAIT
+#       Seconds to wait in Phase 6 for fallback.sh to complete handback.
+#       Must cover: FALLBACK_HANDBACK_STRIKES × FALLBACK_CHECK_INTERVAL + rsync
+#       duration + container start time. At 3 strikes × 30s + ~2min rsync +
+#       ~1min container start: use ≥240s. (default: 300)
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# fallback_test.sh --dry-run
+#     Walk through all 7 phases with output but no iptables changes and no
+#     container starts/stops. ALWAYS run this before a live test.
+#
+# fallback_test.sh
+#     Full live test — real iptables DROP rule, real container lifecycle.
+#     Users will experience a brief service interruption. Run during a
+#     maintenance window.
+#
+# fallback_test.sh --status
+#     Show current fallback state and test timing configuration. No test run.
+#
+# fallback_test.sh --log
+#     Verbose output on every check in every phase.
 #
-# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
-#   fallback_test.sh              — run full test sequence
-#   fallback_test.sh --dry-run    — walk through all phases without changes
-#   fallback_test.sh --status     — show current fallback state and test config
-#   fallback_test.sh --log        — verbose output
 # ==============================================================================================

 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"