feat: rename array orchestrators, add docker_update remainder mode
- Rename array_start.sh → array_started.sh, array_stop.sh → array_stopping.sh
to clarify these are event-driven (array has started/is stopping), not imperative
- Update all references across 9 files (master.conf, user_script_plug-in.sh,
watchdogs, continuous_scripts_status.sh, claude_startup.sh)
- Add --remainder mode to docker_update.sh: updates all running containers
excluding daily containers, weekly sync-window containers (emby + critical-data),
and fallback coverage containers (FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*)
Fallback containers excluded because the remote server owns their version —
independent updates risk writeback incompatibility on handback
- weekly_sync_maintenance.sh calls docker_update.sh --remainder as final step
- git_pull_execute.sh: add safe.directory config to fix dubious ownership error
when running as root on a directory owned by uid 1000
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
co-authored by
Claude Sonnet 4.6
Claude Sonnet 4.6
parent
eaade9a9d4
commit
f16c962ac0
@@ -3,7 +3,7 @@
|
||||
# ============================= Docker Container Stop ==========================================
|
||||
# ==============================================================================================
|
||||
# Stops all running Docker containers one at a time, verifying each is stopped before
|
||||
# moving to the next. Called by array_stop.sh as part of a planned shutdown sequence.
|
||||
# moving to the next. Called by array_stopping.sh as part of a planned shutdown sequence.
|
||||
#
|
||||
# ── STOP SEQUENCE PER CONTAINER ──────────────────────────────────────────────────────────────
|
||||
# 1. docker stop -t 30 (SIGTERM + 30s grace period — docker sends SIGKILL if needed)
|
||||
|
||||
@@ -2,11 +2,26 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Update ==============================================
|
||||
# ==============================================================================================
|
||||
# Two modes — normal (daily) and remainder (weekly).
|
||||
#
|
||||
# ── NORMAL MODE (daily) ───────────────────────────────────────────────────────────────────────
|
||||
# Pulls the latest image for each container in HOST*_DAILY_RESTART_CONTAINERS.
|
||||
# Called by daily_sync_maintenance.sh before docker_daily_restart.sh — containers stay
|
||||
# running during the pull, so there is no extra downtime.
|
||||
#
|
||||
# ── WHY SAME LIST AS DAILY RESTART ───────────────────────────────────────────────────────────
|
||||
# ── REMAINDER MODE (weekly) ───────────────────────────────────────────────────────────────────
|
||||
# Called by weekly_sync_maintenance.sh as the last step.
|
||||
# Updates all currently running containers that are NOT in:
|
||||
# DAILY_RESTART_CONTAINERS — already updated daily
|
||||
# emby + critical-data profiles — already updated inline by the weekly sync window
|
||||
# FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER* — owned by the remote server's update cycle
|
||||
#
|
||||
# Fallback containers are excluded because this server only runs them during a failover.
|
||||
# The remote server is the version owner — if remainder updates them independently and a
|
||||
# handback writeback occurs, the remote's older version may not handle the newer data.
|
||||
# This catches everything local-only (Organizr, AdGuard, etc.) once a week.
|
||||
#
|
||||
# ── WHY SAME LIST AS DAILY RESTART (normal mode) ─────────────────────────────────────────────
|
||||
# Containers that restart daily (auth stack: Authelia, NPM, Mariadb, Redis, etc.) are
|
||||
# exactly the containers that benefit from staying current. Reusing DAILY_RESTART_CONTAINERS
|
||||
# means no second list to maintain — add/remove a container once and both update + restart
|
||||
@@ -16,18 +31,21 @@
|
||||
# docker pull <image> — fetches the latest digest from the registry
|
||||
# Old vs new image ID comparison — distinguishes "updated" from "already current"
|
||||
# Containers keep running — pull does not affect the live container
|
||||
# docker_daily_restart.sh runs after — containers restart on the fresh image
|
||||
# docker_daily_restart.sh runs after (normal mode) — containers restart on the fresh image
|
||||
#
|
||||
# ── TOGGLE ────────────────────────────────────────────────────────────────────────────────────
|
||||
# DAILY_CONTAINER_UPDATES=false in master.conf — skips all pulls, exits cleanly
|
||||
# DAILY_CONTAINER_UPDATES=false in master.conf — skips normal mode, exits cleanly
|
||||
# docker_daily_restart.sh still runs regardless — update and restart are independent
|
||||
# Remainder mode has no toggle — exclude it from WEEKLY_MAINTENANCE_SCRIPTS to disable
|
||||
#
|
||||
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
|
||||
# master.conf: DAILY_CONTAINER_UPDATES — enable/disable (default: true)
|
||||
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update
|
||||
# master.conf: DAILY_CONTAINER_UPDATES — enable/disable normal mode (default: true)
|
||||
# master_host*.conf: HOST*_DAILY_RESTART_CONTAINERS — containers to update (normal mode)
|
||||
# master.conf: PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data] — remainder exclusions
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# docker_update.sh — normal run
|
||||
# docker_update.sh — normal mode: update DAILY_RESTART_CONTAINERS
|
||||
# docker_update.sh --remainder — remainder mode: update all except daily + weekly sync containers
|
||||
# docker_update.sh --dry-run — show what would be pulled
|
||||
# docker_update.sh --log — verbose output
|
||||
# docker_update.sh --status — show config and exit
|
||||
@@ -37,7 +55,20 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
# Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
|
||||
REMAINDER_MODE=false
|
||||
_filtered_args=()
|
||||
for _arg in "$@"; do
|
||||
if [[ "$_arg" == "--remainder" ]]; then
|
||||
REMAINDER_MODE=true
|
||||
else
|
||||
_filtered_args+=("$_arg")
|
||||
fi
|
||||
done
|
||||
unset _arg
|
||||
|
||||
parse_args "${_filtered_args[@]}"
|
||||
unset _filtered_args
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
@@ -54,15 +85,57 @@ fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
log "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Discovery ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
declare -A _exclude=()
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
|
||||
exit 0
|
||||
# Daily containers — updated by docker_update.sh normal mode
|
||||
for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
|
||||
# Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
|
||||
_weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
|
||||
read -r -a _weekly_arr <<< "$_weekly_str"
|
||||
for _c in "${_weekly_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
unset _weekly_str _weekly_arr
|
||||
|
||||
# Fallback coverage containers — owned by the remote server's update cycle.
|
||||
# This server runs them during failover but should never update them independently.
|
||||
# Updating them here risks version divergence: if remote's writeback after handback
|
||||
# encounters data written by a newer version, it may not handle it correctly.
|
||||
for _tier in 1 2 3 4; do
|
||||
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
|
||||
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
|
||||
for _c in "${_tier_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
done
|
||||
unset _tier _tier_var _tier_arr _c
|
||||
|
||||
mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
|
||||
TARGET_CONTAINERS=()
|
||||
for _c in "${_all_running[@]}"; do
|
||||
[[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
|
||||
done
|
||||
unset _all_running _exclude _c
|
||||
else
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
log "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in master_host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -72,8 +145,14 @@ if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
echo "$ICON_GEAR Mode: $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
|
||||
for _c in "${TARGET_CONTAINERS[@]}"; do echo " $_c"; done
|
||||
else
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
fi
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
@@ -81,12 +160,22 @@ fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
|
||||
|
||||
if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
|
||||
log "No containers to update"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
|
||||
else
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
@@ -95,7 +184,7 @@ UP_TO_DATE=()
|
||||
FAILED=()
|
||||
SKIPPED=()
|
||||
|
||||
for container in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
for container in "${TARGET_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
echo "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
@@ -150,7 +239,11 @@ END=$(date +%s)
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
|
||||
else
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
fi
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
[[ ${#UPDATED[@]} -gt 0 ]] && echo "$ICON_DONE Updated: ${UPDATED[*]}"
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# ================================= Docker Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Two-tier self-healing container monitoring system.
|
||||
# Runs continuously as a background process — started by array_start.sh at array start.
|
||||
# Runs continuously as a background process — started by array_started.sh at array start.
|
||||
# Shuts down cleanly on SIGTERM/SIGINT when array stops.
|
||||
#
|
||||
# ── TIER 1 — STRICT MONITORING ────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -140,7 +140,7 @@ if is_script_running "system_watchdog"; then
|
||||
echo " ⏱️ Interval: ${SYSTEM_WATCHDOG_INTERVAL}s │ Heartbeat every: ${SYSTEM_WATCHDOG_HEARTBEAT_HOURS}hr"
|
||||
else
|
||||
echo " ❌ NOT RUNNING — system_watchdog.sh is not active"
|
||||
echo " Start via: bash Orchestrators/array_start.sh"
|
||||
echo " Start via: bash Orchestrators/array_started.sh"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
@@ -254,7 +254,7 @@ if is_script_running "docker_watchdog"; then
|
||||
echo " ⏱️ Interval: ${DOCKER_WATCHDOG_INTERVAL}s │ Heartbeat every: ${DOCKER_WATCHDOG_HEARTBEAT_HOURS}hr"
|
||||
else
|
||||
echo " ❌ NOT RUNNING — docker_watchdog.sh is not active"
|
||||
echo " Start via: bash Orchestrators/array_start.sh"
|
||||
echo " Start via: bash Orchestrators/array_started.sh"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
@@ -400,7 +400,7 @@ else
|
||||
echo " ⏸️ Disabled — FALLBACK_ENABLED=false in master.conf"
|
||||
else
|
||||
echo " ❌ NOT RUNNING — fallback.sh is not active"
|
||||
echo " Start via: bash Orchestrators/array_start.sh"
|
||||
echo " Start via: bash Orchestrators/array_started.sh"
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
@@ -51,10 +51,10 @@
|
||||
# ARRAY_START_SCRIPTS — ordered list of scripts to launch at array start
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# array_start.sh — normal launch (called by User Scripts at array start)
|
||||
# array_start.sh --dry-run — show what would be launched without launching
|
||||
# array_start.sh --status — show configured scripts and their current state
|
||||
# array_start.sh --log — verbose output per script
|
||||
# array_started.sh — normal launch (called by User Scripts at array start)
|
||||
# array_started.sh --dry-run — show what would be launched without launching
|
||||
# array_started.sh --status — show configured scripts and their current state
|
||||
# array_started.sh --log — verbose output per script
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -17,7 +17,7 @@
|
||||
# Containers last — apps should stay available as long as possible during shutdown prep.
|
||||
#
|
||||
# ── SEQUENTIAL vs BACKGROUND ─────────────────────────────────────────────────────────────────
|
||||
# Unlike array_start.sh, all scripts run in the foreground. Each must complete (pass or fail)
|
||||
# Unlike array_started.sh, all scripts run in the foreground. Each must complete (pass or fail)
|
||||
# before the next starts — a failed stop is noted but does not prevent remaining steps.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
@@ -32,10 +32,10 @@
|
||||
# ARRAY_STOP_SCRIPTS — ordered list of stop scripts to run
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# array_stop.sh — run full stop sequence
|
||||
# array_stop.sh --dry-run — preview without stopping anything
|
||||
# array_stop.sh --status — show configured scripts and exit
|
||||
# array_stop.sh --log — verbose output
|
||||
# array_stopping.sh — run full stop sequence
|
||||
# array_stopping.sh --dry-run — preview without stopping anything
|
||||
# array_stopping.sh --status — show configured scripts and exit
|
||||
# array_stopping.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -14,6 +14,7 @@
|
||||
# 6. Start remote containers — correct order, delayed start respected
|
||||
# 7. Start local containers — correct order, delayed start respected
|
||||
# 8. WEEKLY_MAINTENANCE_SCRIPTS — weekly restarts etc. (docker_weekly_restart.sh)
|
||||
# 9. docker_update.sh --remainder — update all containers not in daily or weekly sync window
|
||||
#
|
||||
# ── WHY WEEKLY NOT NIGHTLY FOR EMBY ──────────────────────────────────────────────────────────
|
||||
# Emby builds a warm image cache on HOST2 throughout the week.
|
||||
@@ -330,6 +331,31 @@ if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
|
||||
done
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Remainder Container Updates ━━━
|
||||
# ==============================================================================================
|
||||
# Updates all running containers not already covered by daily or the weekly sync window.
|
||||
# Runs last — weekly sync-window containers are back up before this pulls their peers.
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Remainder Container Updates ━━━"
|
||||
|
||||
DOCKER_UPDATE_SCRIPT="$SCRIPTS_ROOT/Docker_Essentials/docker_update.sh"
|
||||
if [[ ! -f "$DOCKER_UPDATE_SCRIPT" ]]; then
|
||||
warn "docker_update.sh not found — skipping remainder updates"
|
||||
JOB_FAIL+=("docker_update.sh --remainder")
|
||||
else
|
||||
_remainder_args=("--remainder")
|
||||
[[ "$DRY_RUN" == true ]] && _remainder_args+=("--dry-run")
|
||||
if bash "$DOCKER_UPDATE_SCRIPT" "${_remainder_args[@]}"; then
|
||||
log "Remainder updates complete ✅"
|
||||
JOB_PASS+=("docker_update.sh --remainder")
|
||||
else
|
||||
warn "Remainder updates completed with errors"
|
||||
JOB_FAIL+=("docker_update.sh --remainder")
|
||||
fi
|
||||
unset _remainder_args
|
||||
fi
|
||||
|
||||
WINDOW_END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# claude_startup.sh — set up persistent symlinks and launch Claude
|
||||
# claude_startup.sh --setup — set up only, do not launch (for array_start.sh use)
|
||||
# claude_startup.sh --setup — set up only, do not launch (for array_started.sh use)
|
||||
# ==============================================================================================
|
||||
|
||||
PERSIST_DIR="/mnt/user/appdata/claude-code"
|
||||
@@ -89,7 +89,7 @@ _log "claude v$LATEST ready"
|
||||
|
||||
echo ""
|
||||
|
||||
# ── Setup-only mode (used by array_start.sh or other callers) ─────────────────────────────────
|
||||
# ── Setup-only mode (used by array_started.sh or other callers) ─────────────────────────────────
|
||||
if [[ "$LAUNCH" == false ]]; then
|
||||
_log "Setup complete — run 'claude' to start"
|
||||
echo ""
|
||||
|
||||
@@ -195,6 +195,7 @@ if [[ "$DRY_RUN" == true ]]; then
|
||||
SYNC_SUCCESS=true
|
||||
else
|
||||
mkdir -p "$TARGET_DIR"
|
||||
git config --global --add safe.directory "$TARGET_DIR"
|
||||
cd "$TARGET_DIR" || { error "Cannot cd into $TARGET_DIR"; exit 1; }
|
||||
|
||||
if [[ -d ".git" ]]; then
|
||||
|
||||
+6
-6
@@ -36,7 +36,7 @@
|
||||
# GIT / REPO Gitea repository and SSH settings
|
||||
#
|
||||
# ── ORCHESTRATORS ──────────────────────────────────────────────────────────────────────────
|
||||
# ARRAY START Scripts launched at array start (array_start.sh)
|
||||
# ARRAY START Scripts launched at array start (array_started.sh)
|
||||
# DAILY SYNC MAINTENANCE Job list + media shares (daily_sync_maintenance.sh)
|
||||
# WEEKLY SYNC MAINTENANCE Job list + sync shares + update toggles (weekly_sync_maintenance.sh)
|
||||
# CRITICAL SYNC MAINTENANCE 15-minute jobs + sync shares + partnership check (critical_sync_maintenance.sh)
|
||||
@@ -264,7 +264,7 @@
|
||||
# No changes to orchestrator scripts needed when adding or removing jobs.
|
||||
|
||||
# ━━━ Array Stop ━━━
|
||||
# Scripts run by array_stop.sh for a planned shutdown — stops everything cleanly in order.
|
||||
# Scripts run by array_stopping.sh for a planned shutdown — stops everything cleanly in order.
|
||||
# Run sequentially (foreground) — each must complete before the next starts.
|
||||
# Order matters: user scripts first (prevents new ops), then data movement, then containers.
|
||||
ARRAY_STOP_SCRIPTS=(
|
||||
@@ -275,7 +275,7 @@
|
||||
)
|
||||
|
||||
# ━━━ Array Start ━━━
|
||||
# Scripts launched by array_start.sh when the array comes online.
|
||||
# Scripts launched by array_started.sh when the array comes online.
|
||||
# Launched in order — each as a background process.
|
||||
# One-shot scripts (ramdisk, syslog, fpm, inotify, network) run and exit naturally.
|
||||
# Continuous scripts (watchdogs, failover) run until array stops.
|
||||
@@ -561,7 +561,7 @@
|
||||
# ── FALLBACK ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Mutual container failover between two unRAID servers.
|
||||
# Each server runs Fallback/fallback.sh independently via array_start.sh.
|
||||
# Each server runs Fallback/fallback.sh independently via array_started.sh.
|
||||
# All decisions based on two pings: remote reachable + internet reachable.
|
||||
#
|
||||
# States: NORMAL | FALLBACK | NO_INTERNET | DARK
|
||||
@@ -617,7 +617,7 @@
|
||||
|
||||
# ━━━ Docker Watchdog ━━━
|
||||
# Continuous two-tier self-healing container monitoring.
|
||||
# Started by array_start.sh — runs until array stops.
|
||||
# Started by array_started.sh — runs until array stops.
|
||||
# Re-sources all three conf files each cycle — add/remove containers without restarting watchdog.
|
||||
#
|
||||
# Tier 1 — strict monitoring of explicitly configured containers:
|
||||
@@ -1118,7 +1118,7 @@
|
||||
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Continuous system health monitoring — last line of defense before a crash.
|
||||
# Started by array_start.sh — runs until array stops.
|
||||
# Started by array_started.sh — runs until array stops.
|
||||
# Re-sources all three conf files each cycle — config changes take effect on next cycle.
|
||||
#
|
||||
# ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# ================================= System Watchdog ============================================
|
||||
# ==============================================================================================
|
||||
# Last line of defense — reboots the system cleanly if it is about to become unstable.
|
||||
# Runs continuously as a background process — started by array_start.sh at array start.
|
||||
# Runs continuously as a background process — started by array_started.sh at array start.
|
||||
# Works alongside docker_watchdog.sh which handles container-level healing first.
|
||||
#
|
||||
# ── THREE-TIER RESPONSE SYSTEM ────────────────────────────────────────────────────────────────
|
||||
|
||||
+12
-12
@@ -14,7 +14,7 @@
|
||||
#
|
||||
# You do not need to schedule every script below. The orchestrators cover it all:
|
||||
#
|
||||
# array_start.sh at array start — launches ALL startup scripts in order
|
||||
# array_started.sh at array start — launches ALL startup scripts in order
|
||||
# transcode_management.sh every 3 min — cleanup then manager (order critical)
|
||||
# critical_sync_maintenance.sh every 15 min — auth + Emby dirty sync + partnership
|
||||
# intermediate_sync_maintenance.sh every 4 hours — arr library sync + artwork fetch
|
||||
@@ -61,7 +61,7 @@
|
||||
# v1.7 — cert_monitor.sh added
|
||||
# v1.8 — Monitors/ folder with all monitoring scripts
|
||||
# v1.9 — transcode_management.sh added to Orchestrators/
|
||||
# v2.0 — Major restructure: array_start.sh single entry point, daily/weekly orchestrators,
|
||||
# v2.0 — Major restructure: array_started.sh single entry point, daily/weekly orchestrators,
|
||||
# sunday_morning_coffee_report.sh, docker_network_connect combined,
|
||||
# inotify_tuning.sh, arr path translation, version checking, nuclear mode flags
|
||||
# v2.1 — critical_sync_maintenance.sh, Partnership/ folder, system_tuning_monitor.sh,
|
||||
@@ -112,7 +112,7 @@
|
||||
# docker_watchdog.sh [continuous] two-tier container self-healing watchdog
|
||||
# fallback.sh [continuous] mutual fallback state machine
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_started.sh
|
||||
|
||||
|
||||
# ── TRANSCODE MANAGEMENT ──────────────────────────────────────────────────────────────────────
|
||||
@@ -416,7 +416,7 @@
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# fallback.sh — [continuous] mutual fallback state machine
|
||||
# Started by array_start.sh on both servers independently.
|
||||
# Started by array_started.sh on both servers independently.
|
||||
# Every FALLBACK_CHECK_INTERVAL (120s) pings: remote Tailscale IP + 8.8.8.8
|
||||
# States: NORMAL / FALLBACK / NO_INTERNET / DARK
|
||||
# FALLBACK: starts remote containers in tiers across 24 hours:
|
||||
@@ -510,7 +510,7 @@
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# docker_watchdog.sh — [continuous] two-tier container self-healing monitor
|
||||
# Started by array_start.sh. Every DOCKER_WATCHDOG_INTERVAL (900s = 15min).
|
||||
# Started by array_started.sh. Every DOCKER_WATCHDOG_INTERVAL (900s = 15min).
|
||||
#
|
||||
# Tier 1 — explicit per-container (configured in master_host*.conf):
|
||||
# Memory hard limits: immediate restart when exceeded — no strikes, no waiting
|
||||
@@ -577,7 +577,7 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Docker_Essentials/docker_weekly_restart.sh
|
||||
|
||||
# docker_network_connect.sh — ensure custom networks exist and containers are connected
|
||||
# Called by array_start.sh. Idempotent — silent when everything is already correct.
|
||||
# Called by array_started.sh. Idempotent — silent when everything is already correct.
|
||||
# Network missing → create (bridge driver, auto-assigned subnet) → notify.
|
||||
# Network creation should only happen after an unRAID update wiped networks — notify tells you.
|
||||
# Configured via HOST*_NETWORK_CONNECT_NETWORKS and HOST*_NETWORK_CONNECT_CONTAINERS.
|
||||
@@ -637,7 +637,7 @@
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# system_watchdog.sh — [continuous] three-tier server stability last-resort watchdog
|
||||
# Started by array_start.sh. Every SYSTEM_WATCHDOG_INTERVAL (300s = 5min).
|
||||
# Started by array_started.sh. Every SYSTEM_WATCHDOG_INTERVAL (300s = 5min).
|
||||
# All 18 checks independently toggleable per host in master_host*.conf.
|
||||
#
|
||||
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
|
||||
@@ -664,7 +664,7 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh --dry-run
|
||||
|
||||
# inotify_tuning.sh — raise Linux inotify kernel limits at array start
|
||||
# Called by array_start.sh FIRST — must run before containers start (they inherit limits).
|
||||
# Called by array_started.sh FIRST — must run before containers start (they inherit limits).
|
||||
# Settings reset on each reboot — script reapplies on every array start. Idempotent.
|
||||
#
|
||||
# Limits set:
|
||||
@@ -679,7 +679,7 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/inotify_tuning.sh
|
||||
|
||||
# php_fpm_max_children.sh — set PHP-FPM pm.max_children at array start
|
||||
# Called by array_start.sh. Default is 4-8 workers — inadequate for a busy multi-user server.
|
||||
# Called by array_started.sh. Default is 4-8 workers — inadequate for a busy multi-user server.
|
||||
# Sets PHP_MAX_CHILDREN (250). 250 × ~2MB idle = ~500MB. Acceptable on 64GB+.
|
||||
# Symptom of saturation: WebGUI slow, settings saves hang, container UI starts timeout.
|
||||
# Resets on each reboot — reapplied at array start. Idempotent: silent when already correct.
|
||||
@@ -688,7 +688,7 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/php_fpm_max_children.sh
|
||||
|
||||
# docker_syslog_filter.sh — suppress Docker veth/docker0 interface log noise
|
||||
# Called by array_start.sh before containers start. Creates rsyslog drop rule.
|
||||
# Called by array_started.sh before containers start. Creates rsyslog drop rule.
|
||||
# Without this: 50+ containers at array start = 200-400 lines of kernel veth messages.
|
||||
# Real events (mount failures, permission errors) are invisible in that noise.
|
||||
# Idempotent: compares expected filter content exactly — only writes when changed.
|
||||
@@ -895,7 +895,7 @@
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# ramdisk_setup.sh — create tmpfs ramdisk and transcode symlink at array start
|
||||
# Called by array_start.sh. MUST run before Emby starts.
|
||||
# Called by array_started.sh. MUST run before Emby starts.
|
||||
# Idempotent: if ramdisk already mounted → report status and exit cleanly, do not remount.
|
||||
#
|
||||
# Creates:
|
||||
@@ -1198,7 +1198,7 @@
|
||||
# ==============================================================================================
|
||||
#
|
||||
# At Startup of Array:
|
||||
# Orchestrators/array_start.sh (single entry — handles everything)
|
||||
# Orchestrators/array_started.sh (single entry — handles everything)
|
||||
#
|
||||
# */3 * * * * every 3 minutes:
|
||||
# Orchestrators/transcode_management.sh
|
||||
|
||||
Reference in New Issue
Block a user