feat: rename array orchestrators, add docker_update remainder mode

- Rename array_start.sh → array_started.sh, array_stop.sh → array_stopping.sh
  to clarify these are event-driven (array has started/is stopping), not imperative
- Update all references across 9 files (master.conf, user_script_plug-in.sh,
  watchdogs, continuous_scripts_status.sh, claude_startup.sh)
- Add --remainder mode to docker_update.sh: updates all running containers
  excluding daily containers, weekly sync-window containers (emby + critical-data),
  and fallback coverage containers (FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*)
  Fallback containers excluded because the remote server owns their version —
  independent updates risk writeback incompatibility on handback
- weekly_sync_maintenance.sh calls docker_update.sh --remainder as final step
- git_pull_execute.sh: add safe.directory config to fix dubious ownership error
  when running as root on a directory owned by uid 1000

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
2026-05-10 22:12:08 -04:00
co-authored by Claude Sonnet 4.6
parent eaade9a9d4
commit f16c962ac0
12 changed files with 176 additions and 56 deletions
+1 -1
View File
@@ -3,7 +3,7 @@
# ============================= Docker Container Stop ========================================== # ============================= Docker Container Stop ==========================================
# ============================================================================================== # ==============================================================================================
# Stops all running Docker containers one at a time, verifying each is stopped before # Stops all running Docker containers one at a time, verifying each is stopped before
# moving to the next. Called by array_stop.sh as part of a planned shutdown sequence. # moving to the next. Called by array_stopping.sh as part of a planned shutdown sequence.
# #
# ── STOP SEQUENCE PER CONTAINER ────────────────────────────────────────────────────────────── # ── STOP SEQUENCE PER CONTAINER ──────────────────────────────────────────────────────────────
# 1. docker stop -t 30 (SIGTERM + 30s grace period — docker sends SIGKILL if needed) # 1. docker stop -t 30 (SIGTERM + 30s grace period — docker sends SIGKILL if needed)
+114 -21
View File
@@ -2,11 +2,26 @@
# ============================================================================================== # ==============================================================================================
# ================================= Docker Update ============================================== # ================================= Docker Update ==============================================
# ============================================================================================== # ==============================================================================================
# Two modes — normal (daily) and remainder (weekly).
#
# ── NORMAL MODE (daily) ───────────────────────────────────────────────────────────────────────
# Pulls the latest image for each container in HOST*_DAILY_RESTART_CONTAINERS. # Pulls the latest image for each container in HOST*_DAILY_RESTART_CONTAINERS.
# Called by daily_sync_maintenance.sh before docker_daily_restart.sh — containers stay # Called by daily_sync_maintenance.sh before docker_daily_restart.sh — containers stay
# running during the pull, so there is no extra downtime. # running during the pull, so there is no extra downtime.
# #
# ── WHY SAME LIST AS DAILY RESTART ─────────────────────────────────────────────────────────── # ── REMAINDER MODE (weekly) ───────────────────────────────────────────────────────────────────
# Called by weekly_sync_maintenance.sh as the last step.
# Updates all currently running containers that are NOT in:
# DAILY_RESTART_CONTAINERS — already updated daily
# emby + critical-data profiles — already updated inline by the weekly sync window
# FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* — owned by the remote server's update cycle
#
# Fallback containers are excluded because this server only runs them during a failover.
# The remote server is the version owner — if remainder updates them independently and a
# handback writeback occurs, the remote's older version may not handle the newer data.
# This catches everything local-only (Organizr, AdGuard, etc.) once a week.
#
# ── WHY SAME LIST AS DAILY RESTART (normal mode) ─────────────────────────────────────────────
# Containers that restart daily (auth stack: Authelia, NPM, Mariadb, Redis, etc.) are # Containers that restart daily (auth stack: Authelia, NPM, Mariadb, Redis, etc.) are
# exactly the containers that benefit from staying current. Reusing DAILY_RESTART_CONTAINERS # exactly the containers that benefit from staying current. Reusing DAILY_RESTART_CONTAINERS
# means no second list to maintain — add/remove a container once and both update + restart # means no second list to maintain — add/remove a container once and both update + restart
@@ -16,18 +31,21 @@
# docker pull <image> — fetches the latest digest from the registry # docker pull <image> — fetches the latest digest from the registry
# Old vs new image ID comparison — distinguishes "updated" from "already current" # Old vs new image ID comparison — distinguishes "updated" from "already current"
# Containers keep running — pull does not affect the live container # Containers keep running — pull does not affect the live container
# docker_daily_restart.sh runs after — containers restart on the fresh image # docker_daily_restart.sh runs after (normal mode) — containers restart on the fresh image
# #
# ── TOGGLE ──────────────────────────────────────────────────────────────────────────────────── # ── TOGGLE ────────────────────────────────────────────────────────────────────────────────────
# DAILY_CONTAINER_UPDATES=false in master.conf — skips all pulls, exits cleanly # DAILY_CONTAINER_UPDATES=false in master.conf — skips normal mode, exits cleanly
# docker_daily_restart.sh still runs regardless — update and restart are independent # docker_daily_restart.sh still runs regardless — update and restart are independent
# Remainder mode has no toggle — exclude it from WEEKLY_MAINTENANCE_SCRIPTS to disable
# #
# ── CONFIGURATION ───────────────────────────────────────────────────────────────────────────── # ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
# master.conf: DAILY_CONTAINER_UPDATES — enable/disable (default: true) # master.conf: DAILY_CONTAINER_UPDATES — enable/disable normal mode (default: true)
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update # master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update (normal mode)
# master.conf: PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data] — remainder exclusions
# #
# ── USAGE ───────────────────────────────────────────────────────────────────────────────────── # ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_update.sh — normal run # docker_update.sh — normal mode: update DAILY_RESTART_CONTAINERS
# docker_update.sh --remainder — remainder mode: update all except daily + weekly sync containers
# docker_update.sh --dry-run — show what would be pulled # docker_update.sh --dry-run — show what would be pulled
# docker_update.sh --log — verbose output # docker_update.sh --log — verbose output
# docker_update.sh --status — show config and exit # docker_update.sh --status — show config and exit
@@ -37,7 +55,20 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh" source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@" # Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
REMAINDER_MODE=false
_filtered_args=()
for _arg in "$@"; do
if [[ "$_arg" == "--remainder" ]]; then
REMAINDER_MODE=true
else
_filtered_args+=("$_arg")
fi
done
unset _arg
parse_args "${_filtered_args[@]}"
unset _filtered_args
# ============================================================================================== # ==============================================================================================
# ━━━ Setup ━━━ # ━━━ Setup ━━━
@@ -54,15 +85,57 @@ fi
detect_hosts detect_hosts
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then # ==============================================================================================
log "DAILY_CONTAINER_UPDATES=false — skipping container updates" # ━━━ Container Discovery ━━━
exit 0 # ==============================================================================================
fi if [[ "$REMAINDER_MODE" == true ]]; then
declare -A _exclude=()
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then # Daily containers — updated by docker_update.sh normal mode
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update" for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf" [[ -n "$_c" ]] && _exclude["$_c"]=1
exit 0 done
# Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
_weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
read -r -a _weekly_arr <<< "$_weekly_str"
for _c in "${_weekly_arr[@]}"; do
[[ -n "$_c" ]] && _exclude["$_c"]=1
done
unset _weekly_str _weekly_arr
# Fallback coverage containers — owned by the remote server's update cycle.
# This server runs them during failover but should never update them independently.
# Updating them here risks version divergence: if remote's writeback after handback
# encounters data written by a newer version, it may not handle it correctly.
for _tier in 1 2 3 4; do
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
for _c in "${_tier_arr[@]}"; do
[[ -n "$_c" ]] && _exclude["$_c"]=1
done
done
unset _tier _tier_var _tier_arr _c
mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
TARGET_CONTAINERS=()
for _c in "${_all_running[@]}"; do
[[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
done
unset _all_running _exclude _c
else
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
log "DAILY_CONTAINER_UPDATES=false — skipping container updates"
exit 0
fi
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
exit 0
fi
TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
fi fi
# ============================================================================================== # ==============================================================================================
@@ -72,8 +145,14 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo "" echo ""
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━" echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)" echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}" echo "$ICON_GEAR Mode: $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}" if [[ "$REMAINDER_MODE" == true ]]; then
echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
for _c in "${TARGET_CONTAINERS[@]}"; do echo " $_c"; done
else
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
fi
echo "$ICON_GEAR Dry Run: $DRY_RUN" echo "$ICON_GEAR Dry Run: $DRY_RUN"
echo "━━━━━━━━━━━━━━━━━━━━━━━" echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0 exit 0
@@ -81,12 +160,22 @@ fi
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled" [[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
log "No containers to update"
exit 0
fi
# ============================================================================================== # ==============================================================================================
# ━━━ Pull Updates ━━━ # ━━━ Pull Updates ━━━
# ============================================================================================== # ==============================================================================================
echo "" echo ""
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━" if [[ "$REMAINDER_MODE" == true ]]; then
echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}" echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
else
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
fi
echo "" echo ""
START=$(date +%s) START=$(date +%s)
@@ -95,7 +184,7 @@ UP_TO_DATE=()
FAILED=() FAILED=()
SKIPPED=() SKIPPED=()
for container in "${DAILY_RESTART_CONTAINERS[@]}"; do for container in "${TARGET_CONTAINERS[@]}"; do
[[ -z "$container" ]] && continue [[ -z "$container" ]] && continue
echo "━━━ $ICON_CONTAINERS $container ━━━" echo "━━━ $ICON_CONTAINERS $container ━━━"
@@ -150,7 +239,11 @@ END=$(date +%s)
# ============================================================================================== # ==============================================================================================
# ━━━ Summary ━━━ # ━━━ Summary ━━━
# ============================================================================================== # ==============================================================================================
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━" if [[ "$REMAINDER_MODE" == true ]]; then
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
else
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
fi
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)" echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))" echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
[[ ${#UPDATED[@]} -gt 0 ]] && echo "$ICON_DONE Updated: ${UPDATED[*]}" [[ ${#UPDATED[@]} -gt 0 ]] && echo "$ICON_DONE Updated: ${UPDATED[*]}"
+1 -1
View File
@@ -3,7 +3,7 @@
# ================================= Docker Watchdog ============================================ # ================================= Docker Watchdog ============================================
# ============================================================================================== # ==============================================================================================
# Two-tier self-healing container monitoring system. # Two-tier self-healing container monitoring system.
# Runs continuously as a background process — started by array_start.sh at array start. # Runs continuously as a background process — started by array_started.sh at array start.
# Shuts down cleanly on SIGTERM/SIGINT when array stops. # Shuts down cleanly on SIGTERM/SIGINT when array stops.
# #
# ── TIER 1 — STRICT MONITORING ──────────────────────────────────────────────────────────────── # ── TIER 1 — STRICT MONITORING ────────────────────────────────────────────────────────────────
+3 -3
View File
@@ -140,7 +140,7 @@ if is_script_running "system_watchdog"; then
echo " ⏱️ Interval: ${SYSTEM_WATCHDOG_INTERVAL}s │ Heartbeat every: ${SYSTEM_WATCHDOG_HEARTBEAT_HOURS}hr" echo " ⏱️ Interval: ${SYSTEM_WATCHDOG_INTERVAL}s │ Heartbeat every: ${SYSTEM_WATCHDOG_HEARTBEAT_HOURS}hr"
else else
echo " ❌ NOT RUNNING — system_watchdog.sh is not active" echo " ❌ NOT RUNNING — system_watchdog.sh is not active"
echo " Start via: bash Orchestrators/array_start.sh" echo " Start via: bash Orchestrators/array_started.sh"
fi fi
echo "" echo ""
@@ -254,7 +254,7 @@ if is_script_running "docker_watchdog"; then
echo " ⏱️ Interval: ${DOCKER_WATCHDOG_INTERVAL}s │ Heartbeat every: ${DOCKER_WATCHDOG_HEARTBEAT_HOURS}hr" echo " ⏱️ Interval: ${DOCKER_WATCHDOG_INTERVAL}s │ Heartbeat every: ${DOCKER_WATCHDOG_HEARTBEAT_HOURS}hr"
else else
echo " ❌ NOT RUNNING — docker_watchdog.sh is not active" echo " ❌ NOT RUNNING — docker_watchdog.sh is not active"
echo " Start via: bash Orchestrators/array_start.sh" echo " Start via: bash Orchestrators/array_started.sh"
fi fi
echo "" echo ""
@@ -400,7 +400,7 @@ else
echo " ⏸️ Disabled — FALLBACK_ENABLED=false in master.conf" echo " ⏸️ Disabled — FALLBACK_ENABLED=false in master.conf"
else else
echo " ❌ NOT RUNNING — fallback.sh is not active" echo " ❌ NOT RUNNING — fallback.sh is not active"
echo " Start via: bash Orchestrators/array_start.sh" echo " Start via: bash Orchestrators/array_started.sh"
fi fi
fi fi
@@ -51,10 +51,10 @@
# ARRAY_START_SCRIPTS — ordered list of scripts to launch at array start # ARRAY_START_SCRIPTS — ordered list of scripts to launch at array start
# #
# ── USAGE ───────────────────────────────────────────────────────────────────────────────────── # ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# array_start.sh — normal launch (called by User Scripts at array start) # array_started.sh — normal launch (called by User Scripts at array start)
# array_start.sh --dry-run — show what would be launched without launching # array_started.sh --dry-run — show what would be launched without launching
# array_start.sh --status — show configured scripts and their current state # array_started.sh --status — show configured scripts and their current state
# array_start.sh --log — verbose output per script # array_started.sh --log — verbose output per script
# ============================================================================================== # ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -17,7 +17,7 @@
# Containers last — apps should stay available as long as possible during shutdown prep. # Containers last — apps should stay available as long as possible during shutdown prep.
# #
# ── SEQUENTIAL vs BACKGROUND ───────────────────────────────────────────────────────────────── # ── SEQUENTIAL vs BACKGROUND ─────────────────────────────────────────────────────────────────
# Unlike array_start.sh, all scripts run in the foreground. Each must complete (pass or fail) # Unlike array_started.sh, all scripts run in the foreground. Each must complete (pass or fail)
# before the next starts — a failed stop is noted but does not prevent remaining steps. # before the next starts — a failed stop is noted but does not prevent remaining steps.
# #
# ── SAFEGUARDS ──────────────────────────────────────────────────────────────────────────────── # ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
@@ -32,10 +32,10 @@
# ARRAY_STOP_SCRIPTS — ordered list of stop scripts to run # ARRAY_STOP_SCRIPTS — ordered list of stop scripts to run
# #
# ── USAGE ───────────────────────────────────────────────────────────────────────────────────── # ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# array_stop.sh — run full stop sequence # array_stopping.sh — run full stop sequence
# array_stop.sh --dry-run — preview without stopping anything # array_stopping.sh --dry-run — preview without stopping anything
# array_stop.sh --status — show configured scripts and exit # array_stopping.sh --status — show configured scripts and exit
# array_stop.sh --log — verbose output # array_stopping.sh --log — verbose output
# ============================================================================================== # ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+26
View File
@@ -14,6 +14,7 @@
# 6. Start remote containers — correct order, delayed start respected # 6. Start remote containers — correct order, delayed start respected
# 7. Start local containers — correct order, delayed start respected # 7. Start local containers — correct order, delayed start respected
# 8. WEEKLY_MAINTENANCE_SCRIPTS — weekly restarts etc. (docker_weekly_restart.sh) # 8. WEEKLY_MAINTENANCE_SCRIPTS — weekly restarts etc. (docker_weekly_restart.sh)
# 9. docker_update.sh --remainder — update all containers not in daily or weekly sync window
# #
# ── WHY WEEKLY NOT NIGHTLY FOR EMBY ────────────────────────────────────────────────────────── # ── WHY WEEKLY NOT NIGHTLY FOR EMBY ──────────────────────────────────────────────────────────
# Emby builds a warm image cache on HOST2 throughout the week. # Emby builds a warm image cache on HOST2 throughout the week.
@@ -330,6 +331,31 @@ if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
done done
fi fi
# ==============================================================================================
# ━━━ Remainder Container Updates ━━━
# ==============================================================================================
# Updates all running containers not already covered by daily or the weekly sync window.
# Runs last — weekly sync-window containers are back up before this pulls their peers.
echo ""
echo "━━━ $ICON_CONTAINERS Remainder Container Updates ━━━"
DOCKER_UPDATE_SCRIPT="$SCRIPTS_ROOT/Docker_Essentials/docker_update.sh"
if [[ ! -f "$DOCKER_UPDATE_SCRIPT" ]]; then
warn "docker_update.sh not found — skipping remainder updates"
JOB_FAIL+=("docker_update.sh --remainder")
else
_remainder_args=("--remainder")
[[ "$DRY_RUN" == true ]] && _remainder_args+=("--dry-run")
if bash "$DOCKER_UPDATE_SCRIPT" "${_remainder_args[@]}"; then
log "Remainder updates complete ✅"
JOB_PASS+=("docker_update.sh --remainder")
else
warn "Remainder updates completed with errors"
JOB_FAIL+=("docker_update.sh --remainder")
fi
unset _remainder_args
fi
WINDOW_END=$(date +%s) WINDOW_END=$(date +%s)
# ============================================================================================== # ==============================================================================================
+2 -2
View File
@@ -16,7 +16,7 @@
# #
# ── USAGE ───────────────────────────────────────────────────────────────────────────────────── # ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# claude_startup.sh — set up persistent symlinks and launch Claude # claude_startup.sh — set up persistent symlinks and launch Claude
# claude_startup.sh --setup — set up only, do not launch (for array_start.sh use) # claude_startup.sh --setup — set up only, do not launch (for array_started.sh use)
# ============================================================================================== # ==============================================================================================
PERSIST_DIR="/mnt/user/appdata/claude-code" PERSIST_DIR="/mnt/user/appdata/claude-code"
@@ -89,7 +89,7 @@ _log "claude v$LATEST ready"
echo "" echo ""
# ── Setup-only mode (used by array_start.sh or other callers) ───────────────────────────────── # ── Setup-only mode (used by array_started.sh or other callers) ─────────────────────────────────
if [[ "$LAUNCH" == false ]]; then if [[ "$LAUNCH" == false ]]; then
_log "Setup complete — run 'claude' to start" _log "Setup complete — run 'claude' to start"
echo "" echo ""
+1
View File
@@ -195,6 +195,7 @@ if [[ "$DRY_RUN" == true ]]; then
SYNC_SUCCESS=true SYNC_SUCCESS=true
else else
mkdir -p "$TARGET_DIR" mkdir -p "$TARGET_DIR"
git config --global --add safe.directory "$TARGET_DIR"
cd "$TARGET_DIR" || { error "Cannot cd into $TARGET_DIR"; exit 1; } cd "$TARGET_DIR" || { error "Cannot cd into $TARGET_DIR"; exit 1; }
if [[ -d ".git" ]]; then if [[ -d ".git" ]]; then
+6 -6
View File
@@ -36,7 +36,7 @@
# GIT / REPO Gitea repository and SSH settings # GIT / REPO Gitea repository and SSH settings
# #
# ── ORCHESTRATORS ────────────────────────────────────────────────────────────────────────── # ── ORCHESTRATORS ──────────────────────────────────────────────────────────────────────────
# ARRAY START Scripts launched at array start (array_start.sh) # ARRAY START Scripts launched at array start (array_started.sh)
# DAILY SYNC MAINTENANCE Job list + media shares (daily_sync_maintenance.sh) # DAILY SYNC MAINTENANCE Job list + media shares (daily_sync_maintenance.sh)
# WEEKLY SYNC MAINTENANCE Job list + sync shares + update toggles (weekly_sync_maintenance.sh) # WEEKLY SYNC MAINTENANCE Job list + sync shares + update toggles (weekly_sync_maintenance.sh)
# CRITICAL SYNC MAINTENANCE 15-minute jobs + sync shares + partnership check (critical_sync_maintenance.sh) # CRITICAL SYNC MAINTENANCE 15-minute jobs + sync shares + partnership check (critical_sync_maintenance.sh)
@@ -264,7 +264,7 @@
# No changes to orchestrator scripts needed when adding or removing jobs. # No changes to orchestrator scripts needed when adding or removing jobs.
# ━━━ Array Stop ━━━ # ━━━ Array Stop ━━━
# Scripts run by array_stop.sh for a planned shutdown — stops everything cleanly in order. # Scripts run by array_stopping.sh for a planned shutdown — stops everything cleanly in order.
# Run sequentially (foreground) — each must complete before the next starts. # Run sequentially (foreground) — each must complete before the next starts.
# Order matters: user scripts first (prevents new ops), then data movement, then containers. # Order matters: user scripts first (prevents new ops), then data movement, then containers.
ARRAY_STOP_SCRIPTS=( ARRAY_STOP_SCRIPTS=(
@@ -275,7 +275,7 @@
) )
# ━━━ Array Start ━━━ # ━━━ Array Start ━━━
# Scripts launched by array_start.sh when the array comes online. # Scripts launched by array_started.sh when the array comes online.
# Launched in order — each as a background process. # Launched in order — each as a background process.
# One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally. # One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally.
# Continuous scripts (watchdogs, failover) run until array stops. # Continuous scripts (watchdogs, failover) run until array stops.
@@ -561,7 +561,7 @@
# ── FALLBACK ────────────────────────────────────────────────────────────────────────────────── # ── FALLBACK ──────────────────────────────────────────────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
# Mutual container failover between two unRAID servers. # Mutual container failover between two unRAID servers.
# Each server runs Fallback/fallback.sh independently via array_start.sh. # Each server runs Fallback/fallback.sh independently via array_started.sh.
# All decisions based on two pings: remote reachable + internet reachable. # All decisions based on two pings: remote reachable + internet reachable.
# #
# States: NORMAL | FALLBACK | NO_INTERNET | DARK # States: NORMAL | FALLBACK | NO_INTERNET | DARK
@@ -617,7 +617,7 @@
# ━━━ Docker Watchdog ━━━ # ━━━ Docker Watchdog ━━━
# Continuous two-tier self-healing container monitoring. # Continuous two-tier self-healing container monitoring.
# Started by array_start.sh — runs until array stops. # Started by array_started.sh — runs until array stops.
# Re-sources all three conf files each cycle — add/remove containers without restarting watchdog. # Re-sources all three conf files each cycle — add/remove containers without restarting watchdog.
# #
# Tier 1 — strict monitoring of explicitly configured containers: # Tier 1 — strict monitoring of explicitly configured containers:
@@ -1118,7 +1118,7 @@
# ── SYSTEM WATCHDOG ─────────────────────────────────────────────────────────────────────────── # ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
# ============================================================================================== # ==============================================================================================
# Continuous system health monitoring — last line of defense before a crash. # Continuous system health monitoring — last line of defense before a crash.
# Started by array_start.sh — runs until array stops. # Started by array_started.sh — runs until array stops.
# Re-sources all three conf files each cycle — config changes take effect on next cycle. # Re-sources all three conf files each cycle — config changes take effect on next cycle.
# #
# ── THREE-TIER RESPONSE SYSTEM ──────────────────────────────────────────────────────────────── # ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
+1 -1
View File
@@ -3,7 +3,7 @@
# ================================= System Watchdog ============================================ # ================================= System Watchdog ============================================
# ============================================================================================== # ==============================================================================================
# Last line of defense — reboots the system cleanly if it is about to become unstable. # Last line of defense — reboots the system cleanly if it is about to become unstable.
# Runs continuously as a background process — started by array_start.sh at array start. # Runs continuously as a background process — started by array_started.sh at array start.
# Works alongside docker_watchdog.sh which handles container-level healing first. # Works alongside docker_watchdog.sh which handles container-level healing first.
# #
# ── THREE-TIER RESPONSE SYSTEM ──────────────────────────────────────────────────────────────── # ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
+12 -12
View File
@@ -14,7 +14,7 @@
# #
# You do not need to schedule every script below. The orchestrators cover it all: # You do not need to schedule every script below. The orchestrators cover it all:
# #
# array_start.sh at array start — launches ALL startup scripts in order # array_started.sh at array start — launches ALL startup scripts in order
# transcode_management.sh every 3 min — cleanup then manager (order critical) # transcode_management.sh every 3 min — cleanup then manager (order critical)
# critical_sync_maintenance.sh every 15 min — auth + Emby dirty sync + partnership # critical_sync_maintenance.sh every 15 min — auth + Emby dirty sync + partnership
# intermediate_sync_maintenance.sh every 4 hours — arr library sync + artwork fetch # intermediate_sync_maintenance.sh every 4 hours — arr library sync + artwork fetch
@@ -61,7 +61,7 @@
# v1.7 — cert_monitor.sh added # v1.7 — cert_monitor.sh added
# v1.8 — Monitors/ folder with all monitoring scripts # v1.8 — Monitors/ folder with all monitoring scripts
# v1.9 — transcode_management.sh added to Orchestrators/ # v1.9 — transcode_management.sh added to Orchestrators/
# v2.0 — Major restructure: array_start.sh single entry point, daily/weekly orchestrators, # v2.0 — Major restructure: array_started.sh single entry point, daily/weekly orchestrators,
# sunday_morning_coffee_report.sh, docker_network_connect combined, # sunday_morning_coffee_report.sh, docker_network_connect combined,
# inotify_tuning.sh, arr path translation, version checking, nuclear mode flags # inotify_tuning.sh, arr path translation, version checking, nuclear mode flags
# v2.1 — critical_sync_maintenance.sh, Partnership/ folder, system_tuning_monitor.sh, # v2.1 — critical_sync_maintenance.sh, Partnership/ folder, system_tuning_monitor.sh,
@@ -112,7 +112,7 @@
# docker_watchdog.sh [continuous] two-tier container self-healing watchdog # docker_watchdog.sh [continuous] two-tier container self-healing watchdog
# fallback.sh [continuous] mutual fallback state machine # fallback.sh [continuous] mutual fallback state machine
# #
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh # bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_started.sh
# ── TRANSCODE MANAGEMENT ────────────────────────────────────────────────────────────────────── # ── TRANSCODE MANAGEMENT ──────────────────────────────────────────────────────────────────────
@@ -416,7 +416,7 @@
# ────────────────────────────────────────────────────────────────────────────────────────────── # ──────────────────────────────────────────────────────────────────────────────────────────────
# fallback.sh — [continuous] mutual fallback state machine # fallback.sh — [continuous] mutual fallback state machine
# Started by array_start.sh on both servers independently. # Started by array_started.sh on both servers independently.
# Every FALLBACK_CHECK_INTERVAL (120s) pings: remote Tailscale IP + 8.8.8.8 # Every FALLBACK_CHECK_INTERVAL (120s) pings: remote Tailscale IP + 8.8.8.8
# States: NORMAL / FALLBACK / NO_INTERNET / DARK # States: NORMAL / FALLBACK / NO_INTERNET / DARK
# FALLBACK: starts remote containers in tiers across 24 hours: # FALLBACK: starts remote containers in tiers across 24 hours:
@@ -510,7 +510,7 @@
# ────────────────────────────────────────────────────────────────────────────────────────────── # ──────────────────────────────────────────────────────────────────────────────────────────────
# docker_watchdog.sh — [continuous] two-tier container self-healing monitor # docker_watchdog.sh — [continuous] two-tier container self-healing monitor
# Started by array_start.sh. Every DOCKER_WATCHDOG_INTERVAL (900s = 15min). # Started by array_started.sh. Every DOCKER_WATCHDOG_INTERVAL (900s = 15min).
# #
# Tier 1 — explicit per-container (configured in master_host*.conf): # Tier 1 — explicit per-container (configured in master_host*.conf):
# Memory hard limits: immediate restart when exceeded — no strikes, no waiting # Memory hard limits: immediate restart when exceeded — no strikes, no waiting
@@ -577,7 +577,7 @@
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh # bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh
# docker_network_connect.sh — ensure custom networks exist and containers are connected # docker_network_connect.sh — ensure custom networks exist and containers are connected
# Called by array_start.sh. Idempotent — silent when everything is already correct. # Called by array_started.sh. Idempotent — silent when everything is already correct.
# Network missing → create (bridge driver, auto-assigned subnet) → notify. # Network missing → create (bridge driver, auto-assigned subnet) → notify.
# Network creation should only happen after an unRAID update wiped networks — notify tells you. # Network creation should only happen after an unRAID update wiped networks — notify tells you.
# Configured via HOST*_NETWORK_CONNECT_NETWORKS and HOST*_NETWORK_CONNECT_CONTAINERS. # Configured via HOST*_NETWORK_CONNECT_NETWORKS and HOST*_NETWORK_CONNECT_CONTAINERS.
@@ -637,7 +637,7 @@
# ────────────────────────────────────────────────────────────────────────────────────────────── # ──────────────────────────────────────────────────────────────────────────────────────────────
# system_watchdog.sh — [continuous] three-tier server stability last-resort watchdog # system_watchdog.sh — [continuous] three-tier server stability last-resort watchdog
# Started by array_start.sh. Every SYSTEM_WATCHDOG_INTERVAL (300s = 5min). # Started by array_started.sh. Every SYSTEM_WATCHDOG_INTERVAL (300s = 5min).
# All 18 checks independently toggleable per host in master_host*.conf. # All 18 checks independently toggleable per host in master_host*.conf.
# #
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately: # Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
@@ -664,7 +664,7 @@
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
# inotify_tuning.sh — raise Linux inotify kernel limits at array start # inotify_tuning.sh — raise Linux inotify kernel limits at array start
# Called by array_start.sh FIRST — must run before containers start (they inherit limits). # Called by array_started.sh FIRST — must run before containers start (they inherit limits).
# Settings reset on each reboot — script reapplies on every array start. Idempotent. # Settings reset on each reboot — script reapplies on every array start. Idempotent.
# #
# Limits set: # Limits set:
@@ -679,7 +679,7 @@
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/inotify_tuning.sh # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/inotify_tuning.sh
# php_fpm_max_children.sh — set PHP-FPM pm.max_children at array start # php_fpm_max_children.sh — set PHP-FPM pm.max_children at array start
# Called by array_start.sh. Default is 4-8 workers — inadequate for a busy multi-user server. # Called by array_started.sh. Default is 4-8 workers — inadequate for a busy multi-user server.
# Sets PHP_MAX_CHILDREN (250). 250 × ~2MB idle = ~500MB. Acceptable on 64GB+. # Sets PHP_MAX_CHILDREN (250). 250 × ~2MB idle = ~500MB. Acceptable on 64GB+.
# Symptom of saturation: WebGUI slow, settings saves hang, container UI starts timeout. # Symptom of saturation: WebGUI slow, settings saves hang, container UI starts timeout.
# Resets on each reboot — reapplied at array start. Idempotent: silent when already correct. # Resets on each reboot — reapplied at array start. Idempotent: silent when already correct.
@@ -688,7 +688,7 @@
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/php_fpm_max_children.sh # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/php_fpm_max_children.sh
# docker_syslog_filter.sh — suppress Docker veth/docker0 interface log noise # docker_syslog_filter.sh — suppress Docker veth/docker0 interface log noise
# Called by array_start.sh before containers start. Creates rsyslog drop rule. # Called by array_started.sh before containers start. Creates rsyslog drop rule.
# Without this: 50+ containers at array start = 200-400 lines of kernel veth messages. # Without this: 50+ containers at array start = 200-400 lines of kernel veth messages.
# Real events (mount failures, permission errors) are invisible in that noise. # Real events (mount failures, permission errors) are invisible in that noise.
# Idempotent: compares expected filter content exactly — only writes when changed. # Idempotent: compares expected filter content exactly — only writes when changed.
@@ -895,7 +895,7 @@
# ────────────────────────────────────────────────────────────────────────────────────────────── # ──────────────────────────────────────────────────────────────────────────────────────────────
# ramdisk_setup.sh — create tmpfs ramdisk and transcode symlink at array start # ramdisk_setup.sh — create tmpfs ramdisk and transcode symlink at array start
# Called by array_start.sh. MUST run before Emby starts. # Called by array_started.sh. MUST run before Emby starts.
# Idempotent: if ramdisk already mounted → report status and exit cleanly, do not remount. # Idempotent: if ramdisk already mounted → report status and exit cleanly, do not remount.
# #
# Creates: # Creates:
@@ -1198,7 +1198,7 @@
# ============================================================================================== # ==============================================================================================
# #
# At Startup of Array: # At Startup of Array:
# Orchestrators/array_start.sh (single entry — handles everything) # Orchestrators/array_started.sh (single entry — handles everything)
# #
# */3 * * * * every 3 minutes: # */3 * * * * every 3 minutes:
# Orchestrators/transcode_management.sh # Orchestrators/transcode_management.sh