feat: rename array orchestrators, add docker_update remainder mode
- Rename array_start.sh → array_started.sh, array_stop.sh → array_stopping.sh
to clarify these are event-driven (array has started/is stopping), not imperative
- Update all references across 9 files (master.conf, user_script_plug-in.sh,
watchdogs, continuous_scripts_status.sh, claude_startup.sh)
- Add --remainder mode to docker_update.sh: updates all running containers
excluding daily containers, weekly sync-window containers (emby + critical-data),
and fallback coverage containers (FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*)
Fallback containers excluded because the remote server owns their version —
independent updates risk writeback incompatibility on handback
- weekly_sync_maintenance.sh calls docker_update.sh --remainder as final step
- git_pull_execute.sh: add safe.directory config to fix dubious ownership error
when running as root on a directory owned by uid 1000
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
eaade9a9d4
commit
f16c962ac0
@@ -3,7 +3,7 @@
|
||||
# ============================= Docker Container Stop ==========================================
|
||||
# ==============================================================================================
|
||||
# Stops all running Docker containers one at a time, verifying each is stopped before
|
||||
# moving to the next. Called by array_stop.sh as part of a planned shutdown sequence.
|
||||
# moving to the next. Called by array_stopping.sh as part of a planned shutdown sequence.
|
||||
#
|
||||
# ── STOP SEQUENCE PER CONTAINER ──────────────────────────────────────────────────────────────
|
||||
# 1. docker stop -t 30 (SIGTERM + 30s grace period — docker sends SIGKILL if needed)
|
||||
|
||||
@@ -2,11 +2,26 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Update ==============================================
|
||||
# ==============================================================================================
|
||||
# Two modes — normal (daily) and remainder (weekly).
|
||||
#
|
||||
# ── NORMAL MODE (daily) ───────────────────────────────────────────────────────────────────────
|
||||
# Pulls the latest image for each container in HOST*_DAILY_RESTART_CONTAINERS.
|
||||
# Called by daily_sync_maintenance.sh before docker_daily_restart.sh — containers stay
|
||||
# running during the pull, so there is no extra downtime.
|
||||
#
|
||||
# ── WHY SAME LIST AS DAILY RESTART ───────────────────────────────────────────────────────────
|
||||
# ── REMAINDER MODE (weekly) ───────────────────────────────────────────────────────────────────
|
||||
# Called by weekly_sync_maintenance.sh as the last step.
|
||||
# Updates all currently running containers that are NOT in:
|
||||
# DAILY_RESTART_CONTAINERS — already updated daily
|
||||
# emby + critical-data profiles — already updated inline by the weekly sync window
|
||||
# FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* — owned by the remote server's update cycle
|
||||
#
|
||||
# Fallback containers are excluded because this server only runs them during a failover.
|
||||
# The remote server is the version owner — if remainder updates them independently and a
|
||||
# handback writeback occurs, the remote's older version may not handle the newer data.
|
||||
# This catches everything local-only (Organizr, AdGuard, etc.) once a week.
|
||||
#
|
||||
# ── WHY SAME LIST AS DAILY RESTART (normal mode) ─────────────────────────────────────────────
|
||||
# Containers that restart daily (auth stack: Authelia, NPM, Mariadb, Redis, etc.) are
|
||||
# exactly the containers that benefit from staying current. Reusing DAILY_RESTART_CONTAINERS
|
||||
# means no second list to maintain — add/remove a container once and both update + restart
|
||||
@@ -16,18 +31,21 @@
|
||||
# docker pull <image> — fetches the latest digest from the registry
|
||||
# Old vs new image ID comparison — distinguishes "updated" from "already current"
|
||||
# Containers keep running — pull does not affect the live container
|
||||
# docker_daily_restart.sh runs after — containers restart on the fresh image
|
||||
# docker_daily_restart.sh runs after (normal mode) — containers restart on the fresh image
|
||||
#
|
||||
# ── TOGGLE ────────────────────────────────────────────────────────────────────────────────────
|
||||
# DAILY_CONTAINER_UPDATES=false in master.conf — skips all pulls, exits cleanly
|
||||
# DAILY_CONTAINER_UPDATES=false in master.conf — skips normal mode, exits cleanly
|
||||
# docker_daily_restart.sh still runs regardless — update and restart are independent
|
||||
# Remainder mode has no toggle — exclude it from WEEKLY_MAINTENANCE_SCRIPTS to disable
|
||||
#
|
||||
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
|
||||
# master.conf: DAILY_CONTAINER_UPDATES — enable/disable (default: true)
|
||||
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update
|
||||
# master.conf: DAILY_CONTAINER_UPDATES — enable/disable normal mode (default: true)
|
||||
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update (normal mode)
|
||||
# master.conf: PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data] — remainder exclusions
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# docker_update.sh — normal run
|
||||
# docker_update.sh — normal mode: update DAILY_RESTART_CONTAINERS
|
||||
# docker_update.sh --remainder — remainder mode: update all except daily + weekly sync containers
|
||||
# docker_update.sh --dry-run — show what would be pulled
|
||||
# docker_update.sh --log — verbose output
|
||||
# docker_update.sh --status — show config and exit
|
||||
@@ -37,7 +55,20 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
# Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
|
||||
REMAINDER_MODE=false
|
||||
_filtered_args=()
|
||||
for _arg in "$@"; do
|
||||
if [[ "$_arg" == "--remainder" ]]; then
|
||||
REMAINDER_MODE=true
|
||||
else
|
||||
_filtered_args+=("$_arg")
|
||||
fi
|
||||
done
|
||||
unset _arg
|
||||
|
||||
parse_args "${_filtered_args[@]}"
|
||||
unset _filtered_args
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
@@ -54,15 +85,57 @@ fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
log "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Discovery ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
declare -A _exclude=()
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
|
||||
exit 0
|
||||
# Daily containers — updated by docker_update.sh normal mode
|
||||
for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
|
||||
# Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
|
||||
_weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
|
||||
read -r -a _weekly_arr <<< "$_weekly_str"
|
||||
for _c in "${_weekly_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
unset _weekly_str _weekly_arr
|
||||
|
||||
# Fallback coverage containers — owned by the remote server's update cycle.
|
||||
# This server runs them during failover but should never update them independently.
|
||||
# Updating them here risks version divergence: if remote's writeback after handback
|
||||
# encounters data written by a newer version, it may not handle it correctly.
|
||||
for _tier in 1 2 3 4; do
|
||||
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
|
||||
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
|
||||
for _c in "${_tier_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
done
|
||||
unset _tier _tier_var _tier_arr _c
|
||||
|
||||
mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
|
||||
TARGET_CONTAINERS=()
|
||||
for _c in "${_all_running[@]}"; do
|
||||
[[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
|
||||
done
|
||||
unset _all_running _exclude _c
|
||||
else
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
log "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -72,8 +145,14 @@ if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
echo "$ICON_GEAR Mode: $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
|
||||
for _c in "${TARGET_CONTAINERS[@]}"; do echo " $_c"; done
|
||||
else
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
fi
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
@@ -81,12 +160,22 @@ fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
|
||||
|
||||
if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
|
||||
log "No containers to update"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
|
||||
else
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
@@ -95,7 +184,7 @@ UP_TO_DATE=()
|
||||
FAILED=()
|
||||
SKIPPED=()
|
||||
|
||||
for container in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
for container in "${TARGET_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
echo "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
@@ -150,7 +239,11 @@ END=$(date +%s)
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
|
||||
else
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
fi
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
[[ ${#UPDATED[@]} -gt 0 ]] && echo "$ICON_DONE Updated: ${UPDATED[*]}"
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# ================================= Docker Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Two-tier self-healing container monitoring system.
|
||||
# Runs continuously as a background process — started by array_start.sh at array start.
|
||||
# Runs continuously as a background process — started by array_started.sh at array start.
|
||||
# Shuts down cleanly on SIGTERM/SIGINT when array stops.
|
||||
#
|
||||
# ── TIER 1 — STRICT MONITORING ────────────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user