Bring script headers onto the template and close safeguard gaps

Headers claimed protections the code never had, and several destructive paths had no
guard against a collapsed config value.

System_Essentials/clear_logs.sh

@@ -16,9 +16,33 @@
 # SABnzbd) grow fastest — inactive containers typically remain small.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# Two independent passes, each with its own threshold:
+#
+#   System logs — every path in LOG_FILES
+#     size < LOG_MIN_SIZE_MB    → skip, recent diagnostic history is worth keeping
+#     size >= LOG_MIN_SIZE_MB   → truncate in place
+#
+#   Docker logs — /var/lib/docker/containers/**/*-json.log
+#     container name resolved for reporting via docker inspect
+#     size < LOG_DOCKER_MAX_MB  → skip
+#     size >= LOG_DOCKER_MAX_MB → truncate in place
+#     containers directory missing → whole pass skipped, not an error
+#
+# Truncation is always `: > file`, never rm — see Truncate, Never Delete below.
+# Freed bytes are totalled per pass and reported in the summary.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
+# Truncate, Never Delete
+#   Logs are emptied in place, never removed. The writing process keeps its open file
+#   handle and keeps logging; deleting the inode would leave a running daemon writing
+#   to a file nothing can read, and would consume more tmpfs, not less.
+#
 # Size Thresholds, Not Blind Truncation
 #   A 2MB syslog contains useful recent diagnostic history — not worth clearing.
 #   A 500MB Docker log is consuming rootfs and contains mostly noise — clear it.

System_Essentials/conf_cache_restore.sh

@@ -22,6 +22,21 @@
 # Own conf is never in the backup — it's always on disk.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Gates — PARTNERSHIP_ENABLED, and a validated PERSISTENT_CONF_CACHE path
+#   2. No backup directory → exit 0, nothing to restore
+#   3. For each host*.conf in the backup:
+#        own conf              → skip (always on disk)
+#        already in RAM cache  → skip — conf_sync.sh reached the partner, its copy
+#                                is fresher than this one
+#        otherwise             → copy into the RAM cache, mode 600
+#   4. Clear the backup unconditionally — see Remove After Use below
+#
+# Counterpart to conf_cache_save.sh, which writes this backup at array stop.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -40,9 +55,73 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   require_partnership — exits early if PARTNERSHIP_ENABLED=false
-#   detect_hosts()      — determines which conf files belong to partners vs self
-#   No-backup guard     — exits cleanly if PERSISTENT_CONF_CACHE doesn't exist
+# Root Enforcement
+#   Reads the plugin-directory backup and writes the RAM cache.
+#
+# Lock Acquisition
+#   acquire_lock prevents this racing conf_cache_save.sh or the conf cache watchdog
+#   over the same backup directory — this script deletes it at the end.
+#
+# Partnership Gate
+#   require_partnership exits early if PARTNERSHIP_ENABLED=false.
+#
+# Host Detection
+#   detect_hosts() determines which conf files are partner confs and which is our own.
+#
+# Cache Path Sanity Guard
+#   PERSISTENT_CONF_CACHE is validated as an absolute path at least three levels deep
+#   before anything is read or removed. This script ends with rm -rf on that path, and
+#   the directory-exists check alone would not catch a collapsed value — / is a
+#   directory.
+#
+# No-Backup Guard
+#   Exits cleanly if the backup directory does not exist — the normal case when the
+#   partner was reachable at boot.
+#
+# Fresh-Copy Precedence
+#   A conf already present in the RAM cache is never overwritten from the backup.
+#   conf_sync.sh reaching the partner means its copy is current; the backup is by
+#   definition older.
+#
+# Own-Conf Exclusion
+#   Our own conf is never restored from the backup over the live on-disk copy.
+#
+# Credential File Permissions
+#   The RAM cache directory is created 700 and each restored conf written 600 — these
+#   carry partner NPM/lldap passwords and API keys and live under a world-readable /tmp.
+#
+# Dry Run Support
+#   --dry-run reports what would be restored and removed, and changes nothing.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   PERSISTENT_CONF_CACHE
+#       Reboot-surviving backup written by conf_cache_save.sh. Consumed and cleared here.
+#
+#   CONF_RAM_CACHE_DIR
+#       Destination RAM cache (tmpfs, /tmp/.cache/vv/d) that load_config.sh reads
+#       partner vars from.
+#
+#   PARTNERSHIP_ENABLED
+#       Checked via require_partnership().
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# conf_cache_restore.sh
+#     Restore partner confs into the RAM cache, then clear the backup.
+#     Runs at array start, after conf_sync.sh has had its chance.
+#
+# conf_cache_restore.sh --dry-run
+#     Report what would be restored and removed without changing anything
+#
+# conf_cache_restore.sh --log
+#     Verbose per-file output
 #
 # ==============================================================================================
 
@@ -50,11 +129,31 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../load_config.sh"
 
 parse_args "$@"
+
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
+acquire_lock
+
 detect_hosts
 require_partnership
 
 RAM_CACHE="$CONF_RAM_CACHE_DIR"
-SAVE_DIR="$PERSISTENT_CONF_CACHE"
+SAVE_DIR="${PERSISTENT_CONF_CACHE:-}"
+
+# SAVE_DIR is rm -rf'd at the end of this script and is built from ${SCRIPTS_DIR}. If that is
+# ever unset the path collapses toward / — and the -d check below would pass, since / is a
+# directory. Require an absolute path at least three levels deep before touching it.
+_slashes="${SAVE_DIR//[^\/]/}"
+if [[ -z "$SAVE_DIR" || "$SAVE_DIR" != /* || "${#_slashes}" -lt 3 ]]; then
+    error "PERSISTENT_CONF_CACHE is unset or unsafe ('${SAVE_DIR:-unset}') — refusing to restore or clear"
+    notify "conf_cache_restore aborted on $(hostname) — PERSISTENT_CONF_CACHE is '${SAVE_DIR:-unset}'" \
+        "Conf Cache Restore" "warning"
+    exit 1
+fi
+unset _slashes
 
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
 
@@ -80,8 +179,10 @@ for conf in "$SAVE_DIR"/host*.conf; do
         continue
     fi
 
-    mkdir -p "$RAM_CACHE"
-    if cp "$conf" "$RAM_CACHE/$base"; then
+    # Partner confs carry credentials (NPM/lldap passwords, API keys). Default umask would
+    # leave them 644 in a world-readable /tmp path — restrict on the way in, not afterwards.
+    mkdir -p "$RAM_CACHE" && chmod 700 "$RAM_CACHE"
+    if cp "$conf" "$RAM_CACHE/$base" && chmod 600 "$RAM_CACHE/$base"; then
         echo "Restored $base from persistent backup → RAM cache ✅"
         (( restored++ ))
     else

System_Essentials/conf_cache_save.sh

@@ -18,6 +18,19 @@
 # Path adapts to storage mode: $SCRIPTS_DIR/.cache/vv/d (internal or appdata).
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Gates — PARTNERSHIP_ENABLED, and a validated PERSISTENT_CONF_CACHE path
+#   2. No RAM cache present → exit 0, nothing to snapshot
+#   3. For each host*.conf in the RAM cache:
+#        own conf     → skip (always on disk, never needs saving)
+#        partner conf → copy to $PERSISTENT_CONF_CACHE, mode 600
+#
+# Counterpart to conf_cache_restore.sh, which consumes and then clears this backup
+# at the next array start.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -34,9 +47,70 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   require_partnership — exits early if PARTNERSHIP_ENABLED=false
-#   detect_hosts()      — determines which confs to save (partner confs only)
-#   No-cache guard      — exits cleanly if RAM cache is empty or missing
+# Root Enforcement
+#   Writes into $PERSISTENT_CONF_CACHE under the plugin directory.
+#
+# Lock Acquisition
+#   acquire_lock prevents this racing conf_cache_restore.sh or the conf cache
+#   watchdog over the same backup directory.
+#
+# Partnership Gate
+#   require_partnership exits early if PARTNERSHIP_ENABLED=false.
+#
+# Host Detection
+#   detect_hosts() determines which confs are partner confs and which is our own.
+#
+# Cache Path Sanity Guard
+#   PERSISTENT_CONF_CACHE is validated as an absolute path at least three levels
+#   deep before anything is written. It is built from ${SCRIPTS_DIR}; if that were
+#   unset the copy target would collapse to "/host2.conf", dropping partner
+#   passwords and API keys at the filesystem root.
+#
+# No-Cache Guard
+#   Exits cleanly if the RAM cache is missing or empty — nothing to snapshot is a
+#   normal state, not an error.
+#
+# Own-Conf Exclusion
+#   Our own conf is never written into the partner backup. Restoring it later
+#   would overwrite live local config with a stale copy.
+#
+# Credential File Permissions
+#   The backup directory is created 700 and each conf written 600. These files
+#   carry partner NPM/lldap passwords and API keys and must not inherit the
+#   default umask on a path that survives reboot.
+#
+# Dry Run Support
+#   --dry-run reports every file it would write and writes none.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   PERSISTENT_CONF_CACHE
+#       Reboot-surviving destination for the partner conf backup. Built from
+#       ${SCRIPTS_DIR}, so it follows the active storage mode.
+#
+#   CONF_RAM_CACHE_DIR
+#       Source RAM cache (tmpfs, /tmp/.cache/vv/d) populated by conf_sync.sh.
+#
+#   PARTNERSHIP_ENABLED
+#       Checked via require_partnership().
+#
+# ==============================================================================================
+# RUNTIME MODES
+# ==============================================================================================
+#
+# conf_cache_save.sh
+#     Snapshot partner confs from the RAM cache to the persistent backup.
+#     Runs first in ARRAY_STOP_SCRIPTS.
+#
+# conf_cache_save.sh --dry-run
+#     Report what would be saved without writing anything
+#
+# conf_cache_save.sh --log
+#     Verbose per-file output
 #
 # ==============================================================================================
 
@@ -44,11 +118,30 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "$SCRIPT_DIR/../load_config.sh"
 
 parse_args "$@"
+
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
+acquire_lock
+
 detect_hosts
 require_partnership
 
 RAM_CACHE="$CONF_RAM_CACHE_DIR"
-SAVE_DIR="$PERSISTENT_CONF_CACHE"
+SAVE_DIR="${PERSISTENT_CONF_CACHE:-}"
+
+# Credentials get written here. An empty SAVE_DIR would make the cp target "/host2.conf",
+# dropping partner passwords and API keys at the filesystem root.
+_slashes="${SAVE_DIR//[^\/]/}"
+if [[ -z "$SAVE_DIR" || "$SAVE_DIR" != /* || "${#_slashes}" -lt 3 ]]; then
+    error "PERSISTENT_CONF_CACHE is unset or unsafe ('${SAVE_DIR:-unset}') — refusing to save partner confs"
+    notify "conf_cache_save aborted on $(hostname) — PERSISTENT_CONF_CACHE is '${SAVE_DIR:-unset}'" \
+        "Conf Cache Save" "warning"
+    exit 1
+fi
+unset _slashes
 
 [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be written"
 
@@ -69,8 +162,10 @@ for conf in "$RAM_CACHE"/host*.conf; do
         continue
     fi
 
-    mkdir -p "$SAVE_DIR"
-    if cp "$conf" "$SAVE_DIR/$base"; then
+    # Partner confs carry credentials — restrict on write rather than leaving them at the
+    # default umask on a path that survives reboot.
+    mkdir -p "$SAVE_DIR" && chmod 700 "$SAVE_DIR"
+    if cp "$conf" "$SAVE_DIR/$base" && chmod 600 "$SAVE_DIR/$base"; then
         echo "Saved $base → $SAVE_DIR ✅"
         (( saved++ ))
     else

System_Essentials/conf_sync.sh

@@ -25,6 +25,21 @@
 # works whether the remote is in internal or appdata storage mode.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Gates — PARTNERSHIP_ENABLED, CONF_SYNC_ENABLED
+#   2. Cache own conf into the local RAM cache (skipped in --push-only / --pull-only)
+#   3. Per partner:
+#        a. Resolve the partner's own SCRIPTS_DIR by reading their varaverk.cfg over SSH,
+#           so a partner in appdata storage mode is still found
+#        b. Pull  — scp their host*.conf from their disk into our RAM cache
+#        c. Push  — scp our host*.conf into their RAM cache
+#      A partner that fails SSH is counted and skipped; the others still sync.
+#
+# Every file written locally or remotely is restricted to 600, in a 700 directory.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
@@ -42,10 +57,65 @@
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
-#   require_partnership — exits early if PARTNERSHIP_ENABLED=false
-#   detect_hosts()      — partner list for push/pull routing
-#   acquire_lock        — prevents concurrent sync runs
-#   SSH reachability    — partners that fail SSH are skipped, not fatal
+# Root Enforcement
+#   Reads the on-disk conf and writes the RAM cache; SSH/scp run as root.
+#
+# Lock Acquisition
+#   acquire_lock prevents concurrent sync runs writing the same cache files.
+#
+# Partnership Gate
+#   require_partnership exits early if PARTNERSHIP_ENABLED=false.
+#
+# CONF_SYNC_ENABLED Gate
+#   Exits cleanly when disabled, without removing it from the schedule.
+#
+# Host Detection
+#   detect_hosts() builds the partner list used for push/pull routing.
+#
+# SSH Reachability
+#   Partners that fail SSH are counted and skipped, never fatal — one unreachable
+#   partner does not prevent the others from syncing.
+#
+# SSH Timeouts
+#   Every ssh and scp call is wrapped in timeout with ConnectTimeout and BatchMode,
+#   so an unresponsive or password-prompting partner cannot stall the run.
+#
+# Remote Path Discovery
+#   The partner's SCRIPTS_DIR is read from their own varaverk.cfg rather than assumed,
+#   so a partner in appdata storage mode is still found. Falls back to the default
+#   plugin path if the file cannot be read.
+#
+# Credential File Permissions
+#   Cache directories are created 700 and every conf written 600 — on both ends. These
+#   files carry NPM/lldap passwords and API keys, and the cache lives under a
+#   world-readable /tmp path. The pushed copy is chmod'd on the partner too, since our
+#   own credentials land on their disk.
+#
+# Dry Run Support
+#   --dry-run reports every pull and push without transferring anything.
+#
+# ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   CONF_SYNC_ENABLED
+#       Master toggle for conf syncing (default: true)
+#
+#   CONF_RAM_CACHE_DIR
+#       tmpfs cache both ends read partner vars from (/tmp/.cache/vv/d). Cleared every
+#       reboot, which is why conf_cache_save.sh / conf_cache_restore.sh exist.
+#
+#   SSH_KEY
+#       Key used for all partner ssh/scp operations
+#
+#   PARTNERSHIP_ENABLED
+#       Checked via require_partnership()
+#
+# host*.conf
+#
+#   HOST*  — hostnames used to build the partner list via detect_hosts()
 #
 # ==============================================================================================
 # RUNTIME MODES
@@ -73,6 +143,14 @@ for arg in "$@"; do
     esac
 done
 parse_args "${FILTERED_ARGS[@]}"
+
+if [[ "$EUID" -ne 0 ]]; then
+    error "Must be run as root"
+    exit 1
+fi
+
+acquire_lock
+
 detect_hosts
 require_partnership
 
@@ -105,7 +183,10 @@ _remote_scripts_dir() {
 
 # ── Ensure cache dir exists ───────────────────────────────────────────────────
 if [[ "$DRY_RUN" == false ]]; then
-    mkdir -p "$CACHE_DIR"
+    # These conf files carry credentials (NPM/lldap passwords, API keys). The cache lives in
+    # a world-readable /tmp path, so the directory and every file written into it below are
+    # restricted explicitly rather than left at the default umask.
+    mkdir -p "$CACHE_DIR" && chmod 700 "$CACHE_DIR"
 fi
 
 # ── Copy own conf into local cache ───────────────────────────────────────────
@@ -114,8 +195,12 @@ if [[ "$PUSH_ONLY" == false ]] && [[ "$PULL_ONLY" == false ]]; then
         if [[ "$DRY_RUN" == true ]]; then
             warn "DRY RUN — would copy $(basename "$MY_CONF") → $CACHE_DIR/"
         else
-            cp "$MY_CONF" "$CACHE_DIR/${MY_ID,,}.conf" && \
-                echo "Own conf cached ✅" || warn "Failed to cache own conf"
+            if cp "$MY_CONF" "$CACHE_DIR/${MY_ID,,}.conf" && \
+               chmod 600 "$CACHE_DIR/${MY_ID,,}.conf"; then
+                echo "Own conf cached ✅"
+            else
+                warn "Failed to cache own conf"
+            fi
         fi
     else
         warn "Own conf not found: $MY_CONF"
@@ -151,6 +236,7 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
             -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
             "root@${partner_ip}:${remote_conf}" \
             "$CACHE_DIR/${partner_slot}.conf" 2>/dev/null; then
+            chmod 600 "$CACHE_DIR/${partner_slot}.conf" 2>/dev/null
             echo "Pulled ${partner_slot}.conf from $partner_host ✅"
             (( PULLED++ ))
         else
@@ -172,12 +258,16 @@ for host_var in $(compgen -v | grep -E '^HOST[0-9]+$' | sort); do
     # Ensure partner's cache dir exists, then SCP own conf into it
     timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
         -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
-        "root@${partner_ip}" "mkdir -p '$CACHE_DIR'" 2>/dev/null
+        "root@${partner_ip}" "mkdir -p '$CACHE_DIR' && chmod 700 '$CACHE_DIR'" 2>/dev/null
 
     if timeout "$SSH_TIMEOUT" scp -i "$SSH_KEY" \
         -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
         "$MY_CONF" \
         "root@${partner_ip}:${CACHE_DIR}/${MY_ID,,}.conf" 2>/dev/null; then
+        # Our own conf lands on the partner carrying our credentials — restrict it there too.
+        timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
+            -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes -o StrictHostKeyChecking=no \
+            "root@${partner_ip}" "chmod 600 '${CACHE_DIR}/${MY_ID,,}.conf'" 2>/dev/null
         echo "Pushed ${MY_ID,,}.conf to $partner_host ✅"
         (( PUSHED++ ))
     else

System_Essentials/docker_syslog_filter.sh

@@ -16,6 +16,19 @@
 # completely masking real events.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+#   1. Build the expected rsyslog filter content
+#   2. Compare against the file already on disk
+#        identical → exit silently, no write, no rsyslog restart
+#        missing or different → write the filter, then restart rsyslog
+#   3. Verify rsyslog came back up after the restart
+#
+# Runs before any container starts, so the first wave of veth messages at array
+# start is already being filtered.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #

System_Essentials/inotify_tuning.sh

@@ -14,6 +14,26 @@
 # starts requires a docker restart to pick up the new values.
 #
 # ==============================================================================================
+# OPERATIONAL MODEL
+# ==============================================================================================
+#
+# For each of the three limits (instances, watches, queued events):
+#
+#   1. Read the current kernel value via sysctl -n
+#   2. Compare against the configured target
+#        exactly equal → skip, nothing to do
+#        anything else → apply via sysctl -w
+#
+# Note this enforces the configured value exactly, in both directions: a limit currently
+# set HIGHER than the target is lowered back to it. That is deliberate — the conf is the
+# single declared source of truth for these limits — but it means raising a limit by hand
+# will be silently undone at the next array start. Raise the target in master.conf instead.
+#
+# Applied at every array start because these are runtime kernel settings that do not
+# survive a reboot, and must land before containers launch — a container inherits the
+# limits in force at its start, not dynamically.
+#
+# ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #

System_Essentials/rsync_stop.sh

@@ -40,6 +40,35 @@
 #   If remote unreachable → skips remote cleanly, logs warning.
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Stop the Transfer, Not the Schedule
+#   The default mode kills only the rsync subprocess and lets the orchestrator notice
+#   the exit and wind down on its own. Killing the orchestrator too would abandon the
+#   remaining shares silently; letting it finish its own loop keeps the schedule honest
+#   about what ran and what did not.
+#
+# Exact-Name Process Matching
+#   Targets are found with pgrep -x rsync — exact process name, never a pattern match
+#   against a command line. A loose pattern on a box running arbitrary containers could
+#   match something that merely mentions rsync in its arguments.
+#
+# Liveness Checked Before Every Signal
+#   kill -0 confirms a PID is still alive immediately before signalling it. PIDs are
+#   reused, and a transfer that exited on its own between discovery and signalling must
+#   not have its number sent a kill.
+#
+# Interrupting Is Safe by Construction
+#   rsync runs with --partial, so a killed transfer resumes rather than restarting.
+#   That is what makes stopping mid-sync a routine operation rather than a costly one.
+#
+# Clean Up What the Interruption Left
+#   A killed rsync leaves its lock file behind and may leave profile containers stopped.
+#   Both are cleared afterwards, so the next scheduled run is not blocked by a lock
+#   whose owner no longer exists.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
@@ -61,6 +90,31 @@
 #   Remote containers deferred to docker_watchdog.sh.
 #
 # ==============================================================================================
+# CONFIGURATION
+# ==============================================================================================
+#
+# master.conf
+#
+#   LOCK_DIR
+#       Directory holding rsync and orchestrator lock files. Scanned after a kill to
+#       clear locks whose owning PID is gone.
+#
+#   DOCKER_TIMEOUT
+#       Timeout applied to the docker calls used when recovering containers a killed
+#       rsync left stopped.
+#
+#   PROFILE_CRITICAL_CONTAINER_NAMES
+#       Per-profile container lists — used to work out which containers an interrupted
+#       profile sync had stopped and therefore needs restarting.
+#
+#   SSH_KEY / SSH timeouts
+#       Used to reach the partner when stopping its rsync as well.
+#
+# host*.conf
+#
+#   HOST*  — resolved via detect_hosts() for MY_ID and remote routing
+#
+# ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
 #

System_Essentials/server_reboot.sh

@@ -37,6 +37,31 @@
 #   After REBOOT_VM_WAIT seconds, libvirt stops anyway — reboot takes priority.
 #
 # ==============================================================================================
+# DESIGN PRINCIPLES
+# ==============================================================================================
+#
+# Warn, Do Not Block
+#   Pre-flight reports what is in flight — rsync running, mover running — and proceeds
+#   anyway. This is an operator-invoked tool: the person running it has already decided
+#   to reboot, and refusing would just push them to /sbin/reboot with no warning, no
+#   wall message and no clean array stop. The warnings name the specific script to run
+#   first (rsync_stop.sh, mover_stop.sh) so the safer path is the obvious one.
+#
+# Announce Before Acting
+#   A wall message and a notification go out REBOOT_SLEEP seconds ahead, both naming
+#   which host is rebooting and why. On a two-server setup "the server is rebooting" is
+#   ambiguous and therefore useless.
+#
+# Clean Array Stop First
+#   The reboot routes through the normal array stop sequence rather than calling
+#   /sbin/reboot directly. Array stop failures are reported and the reboot continues —
+#   an already-committed reboot should not be abandoned halfway, leaving services down
+#   and the machine still up.
+#
+# Flush Before Cutting Power
+#   sync runs immediately before /sbin/reboot so buffered writes reach disk.
+#
+# ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #