Audit every bash script in these directories against its own header documentation. For each script, read the full file and check:
1. Does PURPOSE match what the code actually does?
2. Does CONFIGURATION list all config vars the script actually uses (missing vars, or vars listed but never referenced)?
3. Does RUNTIME MODES / flags match what's actually parsed (missing flags, or flags documented but not implemented)?
4. Are there OPERATIONAL SAFEGUARDS documented that aren't in the code?
5. Anything the code does that the header doesn't mention?

Scripts to audit:
- /boot/config/plugins/varaverk/Deployment/conf_upgrade.sh
- /boot/config/plugins/varaverk/Deployment/deploy.sh
- /boot/config/plugins/varaverk/Docker_Essentials/docker_container_stop.sh
- /boot/config/plugins/varaverk/Docker_Essentials/docker_daily_restart.sh
- /boot/config/plugins/varaverk/Docker_Essentials/docker_network_connect.sh
- /boot/config/plugins/varaverk/Docker_Essentials/docker_update.sh
- /boot/config/plugins/varaverk/Docker_Essentials/docker_update_remaining.sh
- /boot/config/plugins/varaverk/Docker_Essentials/docker_weekly_restart.sh
- /boot/config/plugins/varaverk/Docker_Essentials/downloaders_reset.sh

Report ONLY real discrepancies — not style opinions. Format: filename, then a short bullet list of issues. If a script's header matches its code exactly, just write "✅ filename". Be specific: quote the header claim and what the code actually does differently.


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled cloud agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring cloud agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


I'll audit each bash script by reading its full content and comparing the header documentation with the actual code. Let me start by reading all the scripts.


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= conf_upgrade.sh ================================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Merges a new conf template into an existing user conf while preserving every
9	# value the user has already set. Run manually when the conf schema changes
10	# between versions — adds new keys, removes deprecated ones, and keeps the
11	# structure of the new template exactly.
12	#
13	#   Keys in template only  → ADDED   (placeholder/default — user fills in once)
14	#   Keys in target only    → REMOVED (deprecated in new version)
15	#   Keys in both           → KEPT    (target's value always wins, template ignored)
16	#   Comments / blank lines → always from template (structure follows new version)
17	#
18	# Supports all conf variable patterns: simple scalars, indexed arrays, and
19	# associative arrays (declare -A).
20	#
21	# ==============================================================================================
22	# OPERATIONAL SAFEGUARDS
23	# ==============================================================================================
24	#
25	# Dry-run Mode
26	#   --dry-run prints the full change report (ADDED / REMOVED / KEPT) then exits
27	#   without writing anything. Always preview before applying to production confs.
28	#
29	# Backup Option
30	#   --backup writes a .bak copy of the target before overwriting. Use when
31	#   applying to a conf that has never been upgraded before.
32	#
33	# File Existence Guards
34	#   Both --template and --target are validated before any parsing begins.
35	#   Missing files abort immediately with a clear error.
36	#
37	# Atomic Write
38	#   Merged output is written to a tempfile first, then copied to the target.
39	#   A partial write cannot corrupt the original.
40	#
41	# ==============================================================================================
42	# CONFIGURATION
43	# ==============================================================================================
44	#
45	# No conf vars. All inputs are CLI flags.
46	#
47	#   --template <file>   New version conf file (source of structure and defaults)
48	#   --target   <file>   Existing user conf (source of real values — always preserved)
49	#   --dry-run           Show what would change without writing
50	#   --backup            Write a .bak copy of target before modifying
51	#
52	# ==============================================================================================
53	# RUNTIME MODES
54	# ==============================================================================================
55	#
56	# conf_upgrade.sh --template Configurations/master.conf.template --target Configurations/master.conf --dry-run
57	#     Preview what would be added, removed, and kept — no changes written.
58	#
59	# conf_upgrade.sh --template Configurations/master.conf.template --target Configurations/master.conf --backup
60	#     Apply the upgrade, writing a .bak first.
61	#
62	# conf_upgrade.sh --template Configurations/master.conf.template --target Configurations/master.conf
63	#     Apply the upgrade in-place with no backup.
64	#
65	# ==============================================================================================
66
67	set -uo pipefail
68
69	# ── Arguments ────────────────────────────────────────────────────────────────────────────────
70
71	TEMPLATE=""
72	TARGET=""
73	DRY_RUN=false
74	BACKUP=false
75
76	while [[ $# -gt 0 ]]; do
77	    case "$1" in
78	        --template) TEMPLATE="$2"; shift 2 ;;
79	        --target)   TARGET="$2";   shift 2 ;;
80	        --dry-run)  DRY_RUN=true;  shift   ;;
81	        --backup)   BACKUP=true;   shift   ;;
82	        *) echo "Unknown option: $1" >&2; exit 1 ;;
83	    esac
84	done
85
86	[[ -z "$TEMPLATE" ]] && { echo "Error: --template required" >&2; exit 1; }
87	[[ -z "$TARGET"   ]] && { echo "Error: --target required"   >&2; exit 1; }
88	[[ -f "$TEMPLATE" ]] || { echo "Error: template not found: $TEMPLATE" >&2; exit 1; }
89	[[ -f "$TARGET"   ]] || { echo "Error: target not found: $TARGET"     >&2; exit 1; }
90
91	# ── Parse target → KEY → full definition block ───────────────────────────────────────────────
92
93	declare -A HOST_MAP   # KEY → complete definition line(s) from user's conf
94
95	_parse_target() {
96	    local in_block=false cur_key="" cur_block="" line
97
98	    while IFS= read -r line || [[ -n "$line" ]]; do
99	        if [[ "$in_block" == true ]]; then
100	            cur_block+="$line"$'\n'
101	            # Closing ) — optional trailing whitespace and comment
102	            if [[ "$line" =~ ^[[:space:]]*\)[[:space:]]*(#.*)?$ ]]; then
103	                HOST_MAP["$cur_key"]="$cur_block"
104	                in_block=false; cur_key=""; cur_block=""
105	            fi
106	        else
107	            # declare -A KEY=(
108	            if [[ "$line" =~ ^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\( ]]; then
109	                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
110	            # KEY=(
111	            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\( ]]; then
112	                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
113	            # KEY=value  (simple scalar)
114	            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*= ]]; then
115	                HOST_MAP["${BASH_REMATCH[1]}"]="$line"$'\n'
116	            fi
117	        fi
118	    done < "$TARGET"
119	}
120
121	# ── Walk template — collect stats (must run in current shell so arrays persist) ──────────────
122
123	declare -a ADDED=() KEPT=() REMOVED=()
124	declare -A TMPL_SEEN=()
125
126	_collect_stats() {
127	    local in_block=false cur_key="" line
128
129	    while IFS= read -r line || [[ -n "$line" ]]; do
130	        if [[ "$in_block" == true ]]; then
131	            if [[ "$line" =~ ^[[:space:]]*\)[[:space:]]*(#.*)?$ ]]; then
132	                in_block=false
133	                TMPL_SEEN["$cur_key"]=1
134	                if [[ -n "${HOST_MAP[$cur_key]+_}" ]]; then KEPT+=("$cur_key")
135	                else                                         ADDED+=("$cur_key"); fi
136	                cur_key=""
137	            fi
138	        else
139	            if [[ "$line" =~ ^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\( ]]; then
140	                cur_key="${BASH_REMATCH[1]}"; in_block=true
141	            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\( ]]; then
142	                cur_key="${BASH_REMATCH[1]}"; in_block=true
143	            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*= ]]; then
144	                local k="${BASH_REMATCH[1]}"
145	                TMPL_SEEN["$k"]=1
146	                if [[ -n "${HOST_MAP[$k]+_}" ]]; then KEPT+=("$k")
147	                else                                   ADDED+=("$k"); fi
148	            fi
149	        fi
150	    done < "$TEMPLATE"
151
152	    for key in "${!HOST_MAP[@]}"; do
153	        [[ -z "${TMPL_SEEN[$key]+_}" ]] && REMOVED+=("$key")
154	    done
155	}
156
157	# ── Walk template — write merged output ──────────────────────────────────────────────────────
158	# Runs in a subshell (stdout redirected) — array mutations are intentionally discarded here.
159
160	_write_merged() {
161	    local in_block=false cur_key="" cur_block="" line
162
163	    while IFS= read -r line || [[ -n "$line" ]]; do
164	        if [[ "$in_block" == true ]]; then
165	            cur_block+="$line"$'\n'
166	            if [[ "$line" =~ ^[[:space:]]*\)[[:space:]]*(#.*)?$ ]]; then
167	                in_block=false
168	                if [[ -n "${HOST_MAP[$cur_key]+_}" ]]; then printf '%s' "${HOST_MAP[$cur_key]}"
169	                else                                        printf '%s' "$cur_block"; fi
170	                cur_key=""; cur_block=""
171	            fi
172	        else
173	            if [[ "$line" =~ ^[[:space:]]*declare[[:space:]]+-[a-zA-Z]+[[:space:]]+([A-Z0-9_]+)[[:space:]]*=\( ]]; then
174	                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
175	            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*=\( ]]; then
176	                cur_key="${BASH_REMATCH[1]}"; in_block=true; cur_block="$line"$'\n'
177	            elif [[ "$line" =~ ^[[:space:]]*([A-Z0-9_]+)[[:space:]]*= ]]; then
178	                local k="${BASH_REMATCH[1]}"
179	                if [[ -n "${HOST_MAP[$k]+_}" ]]; then printf '%s' "${HOST_MAP[$k]}"
180	                else                                  printf '%s\n' "$line"; fi
181	            else
182	                printf '%s\n' "$line"
183	            fi
184	        fi
185	    done < "$TEMPLATE"
186	}
187
188	# ── Run ───────────────────────────────────────────────────────────────────────────────────────
189
190	_parse_target
191	_collect_stats
192
193	# ── Report ────────────────────────────────────────────────────────────────────────────────────
194
195	TARGET_NAME="$(basename "$TARGET")"
196	echo ""
197	echo "── conf_upgrade: $TARGET_NAME ──────────────────────────────────────────"
198
199	if [[ ${#ADDED[@]} -gt 0 ]]; then
200	    echo "  ADDED   (new — fill in your values where needed):"
201	    for k in "${ADDED[@]}"; do echo "    + $k"; done
202	fi
203
204	if [[ ${#REMOVED[@]} -gt 0 ]]; then
205	    echo "  REMOVED (deprecated — no longer in this version):"
206	    for k in "${REMOVED[@]}"; do echo "    - $k"; done
207	fi
208
209	echo "  KEPT    ${#KEPT[@]} existing vars — your values preserved"
210
211	if [[ ${#ADDED[@]} -eq 0 && ${#REMOVED[@]} -eq 0 ]]; then
212	    echo "  Already up to date — no changes needed."
213	    echo "────────────────────────────────────────────────────────────────────────"
214	    echo ""
215	    exit 0
216	fi
217
218	echo "────────────────────────────────────────────────────────────────────────"
219	echo ""
220
221	# ── Apply ─────────────────────────────────────────────────────────────────────────────────────
222
223	if [[ "$DRY_RUN" == true ]]; then
224	    echo "(dry-run — no changes written)"
225	    exit 0
226	fi
227
228	TMPOUT="$(mktemp)"
229	trap 'rm -f "$TMPOUT"' EXIT
230
231	_write_merged > "$TMPOUT"
232
233	if [[ "$BACKUP" == true ]]; then
234	    cp "$TARGET" "${TARGET}.bak"
235	    echo "Backup: ${TARGET}.bak"
236	fi
237
238	cp "$TMPOUT" "$TARGET"
239	echo "Updated: $TARGET"
240


1	#!/bin/bash
2	# ==============================================================================================
3	# deploy.sh — Sync dev working tree → prod runtime for Plugin PHP and conf files.
4	# ==============================================================================================
5	#
6	# Direct call (interactive):  ! bash Deployment/deploy.sh
7	#   Deploys both Plugin and Configurations with full output.
8	#
9	# Hook call (Claude Code PostToolUse):
10	#   Reads the tool JSON from stdin, deploys only the component that was actually changed.
11	#   Plugin/  → rsync --delete into prod Plugin/unraid/
12	#   Configurations/ → conf_upgrade.sh (adds new keys, preserves prod values)
13	#
14	# ==============================================================================================
15
16	DEV_ROOT="/mnt/cloud-storage/Important Shit/Git/Development/Varaverk"
17	PROD_ROOT="/boot/config/plugins/varaverk"
18	UPGRADE="$PROD_ROOT/Deployment/conf_upgrade.sh"
19	LOG="/tmp/vv_deploy.log"
20
21	# ── Deploy functions ──────────────────────────────────────────────────────────
22
23	deploy_plugin() {
24	    rsync -a --delete --exclude='.git' \
25	        "$DEV_ROOT/Plugin/unraid/" \
26	        "$PROD_ROOT/Plugin/unraid/"
27	}
28
29	deploy_conf() {
30	    for conf in master.conf host1.conf host2.conf; do
31	        tmpl="$DEV_ROOT/Configurations/$conf"
32	        target="$PROD_ROOT/Configurations/$conf"
33	        [[ -f "$tmpl" && -f "$target" ]] && \
34	            "$UPGRADE" --template "$tmpl" --target "$target"
35	    done
36	}
37
38	# ── Entry point ───────────────────────────────────────────────────────────────
39
40	# Read stdin (non-blocking — empty when called directly or via '!')
41	STDIN=$(cat 2>/dev/null || true)
42
43	# Extract file_path from hook JSON if present
44	FILE=$(python3 -c "
45	import json, sys
46	try:
47	    d = json.loads(sys.stdin.read())
48	    print(d.get('tool_input', {}).get('file_path', ''))
49	except:
50	    print('')
51	" <<< "$STDIN" 2>/dev/null || echo "")
52
53	if [[ -z "$FILE" ]]; then
54	    # Direct call (interactive or via !) — deploy everything with output
55	    echo "=== deploy: dev → prod ==="
56	    echo ""
57	    echo "── Plugin ──"
58	    deploy_plugin && echo "  rsync done"
59	    echo ""
60	    echo "── Configurations ──"
61	    deploy_conf
62	    echo ""
63	    echo "=== done ==="
64	    exit 0
65	fi
66
67	# Hook call — deploy only the affected component, log silently
68	{
69	    echo "[$(date '+%H:%M:%S')] hook: $FILE"
70
71	    if [[ "$FILE" == "$DEV_ROOT/Plugin/"* ]]; then
72	        deploy_plugin && echo "  plugin synced"
73	    fi
74
75	    if [[ "$FILE" == "$DEV_ROOT/Configurations/"* ]]; then
76	        deploy_conf && echo "  conf upgraded"
77	    fi
78	} >> "$LOG" 2>&1
79
80	exit 0
81


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Docker Container Stop ==========================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Gracefully stops all running Docker containers in a verified sequential order.
9	#
10	# Called by array_stopping.sh during planned shutdowns, maintenance windows,
11	# and controlled reboot operations.
12	#
13	# The script guarantees each container is fully stopped before moving to the
14	# next one — preventing dependency breakage, partial shutdown states, and
15	# abrupt service termination cascades.
16	#
17	# ==============================================================================================
18	# OPERATIONAL MODEL
19	# ==============================================================================================
20	#
21	# Shutdown is processed one container at a time:
22	#
23	#   1. docker stop -t 30
24	#        → sends SIGTERM with graceful shutdown window
25	#
26	#   2. Verify container state
27	#        → confirm container fully stopped before continuing
28	#
29	#   3. Retry if still running
30	#        → up to RETRY_COUNT attempts
31	#
32	#   4. Escalate to docker kill
33	#        → SIGKILL only after graceful attempts exhausted
34	#
35	#   5. Final verification
36	#        → failure notification if container survives SIGKILL
37	#
38	# The script NEVER advances to the next container until the current one
39	# is confirmed stopped or declared failed.
40	#
41	# ==============================================================================================
42	# DESIGN PRINCIPLES
43	# ==============================================================================================
44	#
45	# Sequential Shutdown
46	#   Containers may depend on upstream services still being available during
47	#   shutdown. Sequential processing reduces dependency severance during stop.
48	#
49	# Graceful First, Forced Last
50	#   SIGTERM is always attempted before SIGKILL. The script never force-kills
51	#   first unless Docker itself escalates internally after timeout expiration.
52	#
53	# Verification Over Assumption
54	#   Docker command success alone is not trusted. Container state is verified
55	#   after every stop attempt.
56	#
57	# Fail Loudly
58	#   Containers that cannot be stopped generate notifications and non-zero exit
59	#   status so orchestrators know shutdown integrity was compromised.
60	#
61	# ==============================================================================================
62	# OPERATIONAL SAFEGUARDS
63	# ==============================================================================================
64	#
65	# Root Enforcement
66	#   Docker operations require root privileges.
67	#
68	# Docker Presence Check
69	#   Verifies docker binary exists before execution.
70	#
71	# Lock Acquisition
72	#   acquire_lock prevents overlapping runs (e.g. array_stopping firing twice).
73	#
74	# Timeout Protection
75	#   All docker commands wrapped in timeout protection to prevent daemon hangs
76	#   from stalling shutdown indefinitely.
77	#
78	# Retry Escalation
79	#   Graceful retries occur before SIGKILL escalation.
80	#
81	# Per-Container Validation
82	#   Every container state verified before progressing to the next.
83	#
84	# Deterministic Ordering
85	#   Running container list sorted before processing for stable execution order.
86	#
87	# Failure Notification
88	#   Containers surviving SIGKILL trigger operator notification.
89	#
90	# ==============================================================================================
91	# CONFIGURATION
92	# ==============================================================================================
93	#
94	# master.conf
95	#
96	#   RETRY_COUNT
97	#       Graceful retry attempts before force-kill escalation
98	#
99	#   SLEEP
100	#       Delay in seconds between retry attempts
101	#
102	# ==============================================================================================
103	# RUNTIME MODES
104	# ==============================================================================================
105	#
106	# docker_container_stop.sh
107	#     Stop all running containers in verified sequential order
108	#
109	# docker_container_stop.sh --dry-run
110	#     Preview shutdown actions without stopping containers
111	#
112	# docker_container_stop.sh --status
113	#     Show currently running containers and configuration state
114	#
115	# docker_container_stop.sh --log
116	#     Verbose per-container execution logging
117	#
118	# ==============================================================================================
119
120	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
121
122	source "$SCRIPT_DIR/../load_config.sh"
123
124	parse_args "$@"
125
126	# ==============================================================================================
127	# ━━━ Setup ━━━
128	# ==============================================================================================
129	if [[ "$EUID" -ne 0 ]]; then
130	    error "Must be run as root"
131	    exit 1
132	fi
133
134	acquire_lock
135
136	if ! command -v docker &>/dev/null; then
137	    error "Docker command not found"
138	    exit 1
139	fi
140
141	detect_hosts
142
143	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be stopped"
144
145	DOCKER_TIMEOUT=30
146	DOCKER_STOP_TIMEOUT=30    # grace period for SIGTERM before docker sends SIGKILL internally
147	_RETRY_COUNT="${RETRY_COUNT:-3}"
148
149	log "$ICON_GEAR Config: retries=$_RETRY_COUNT sleep=${SLEEP:-5}s grace=${DOCKER_STOP_TIMEOUT}s cmd-timeout=${DOCKER_TIMEOUT}s"
150
151	# ==============================================================================================
152	# ━━━ Status ━━━
153	# ==============================================================================================
154	if [[ "$SHOW_STATUS" == true ]]; then
155	    mapfile -t RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
156	    echo ""
157	    echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
158	    echo "$ICON_HOST       Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
159	    echo "$ICON_CONTAINERS Running:   ${#RUNNING[@]} container(s)"
160	    echo "$ICON_RETRY      Retries:   $_RETRY_COUNT"
161	    for c in "${RUNNING[@]}"; do echo "  $ICON_RUNNING $c"; done
162	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
163	    exit 0
164	fi
165
166	# ==============================================================================================
167	# ━━━ Stop Containers ━━━
168	# ==============================================================================================
169	mapfile -t RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
170
171	echo ""
172	echo "━━━ $ICON_CONTAINERS Docker Container Stop — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
173
174	if [[ ${#RUNNING[@]} -eq 0 ]]; then
175	    echo "No running containers — nothing to do"
176	    exit 0
177	fi
178
179	echo "$ICON_CONTAINERS Containers: ${#RUNNING[@]} running"
180	log "$ICON_CONTAINERS Queue: ${RUNNING[*]}"
181	echo ""
182
183	START=$(date +%s)
184	STOPPED=()
185	FAILED=()
186
187	for container in "${RUNNING[@]}"; do
188	    [[ -z "$container" ]] && continue
189	    local c_start
190	    c_start=$(date +%s)
191	    local c_image
192	    c_image=$(docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
193	    log "━━━ $ICON_CONTAINERS $container ($c_image) ━━━"
194
195	    if [[ "$DRY_RUN" == true ]]; then
196	        warn "DRY RUN — would stop $container"
197	        STOPPED+=("$container")
198	        continue
199	    fi
200
201	    attempt=1
202	    success=false
203
204	    while [[ "$attempt" -le "$_RETRY_COUNT" ]]; do
205	        log "$ICON_RETRY Attempt $attempt of $_RETRY_COUNT — stopping $container..."
206
207	        timeout "$DOCKER_TIMEOUT" docker stop -t "$DOCKER_STOP_TIMEOUT" "$container" \
208	            >/dev/null 2>&1
209
210	        STATE=$(timeout "$DOCKER_TIMEOUT" docker inspect \
211	            -f '{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
212
213	        if [[ "$STATE" != "true" ]]; then
214	            log "$ICON_DONE $container stopped in $(format_duration $(( $(date +%s) - c_start ))) ✅"
215	            STOPPED+=("$container")
216	            success=true
217	            break
218	        fi
219
220	        warn "Attempt $attempt failed — $container still running"
221	        (( attempt++ ))
222	        [[ "$attempt" -le "$_RETRY_COUNT" ]] && sleep "${SLEEP:-5}"
223	    done
224
225	    if [[ "$success" == false ]]; then
226	        warn "$container — retries exhausted, force-killing with SIGKILL..."
227	        timeout "$DOCKER_TIMEOUT" docker kill "$container" >/dev/null 2>&1 || true
228
229	        STATE=$(timeout "$DOCKER_TIMEOUT" docker inspect \
230	            -f '{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
231
232	        if [[ "$STATE" != "true" ]]; then
233	            warn "$container force-killed ✅"
234	            STOPPED+=("$container")
235	        else
236	            error "$container still running after SIGKILL — manual intervention needed"
237	            notify "$container could not be stopped on $(hostname) ($MY_ID)" \
238	                "Docker Container Stop" "warning"
239	            FAILED+=("$container")
240	        fi
241	    fi
242	done
243
244	END=$(date +%s)
245
246	# ==============================================================================================
247	# ━━━ Summary ━━━
248	# ==============================================================================================
249	echo "━━━━━ $ICON_SUMMARY DOCKER CONTAINER STOP SUMMARY ━━━━━"
250	echo "$ICON_HOST       Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
251	echo "$ICON_TIME       Duration:  $(format_duration $(( END - START )))"
252	echo "$ICON_CONTAINERS Scope:     ${#RUNNING[@]} running → ${#STOPPED[@]} stopped"
253	[[ ${#STOPPED[@]} -gt 0 ]] && log "$ICON_DONE  Stopped:   ${STOPPED[*]}"
254	[[ ${#FAILED[@]} -gt 0 ]]  && echo "$ICON_ERROR Failed:    ${FAILED[*]}"
255
256	if [[ "$DRY_RUN" == true ]]; then
257	    warn "DRY RUN — no containers stopped"
258	elif [[ ${#FAILED[@]} -eq 0 ]]; then
259	    echo "$ICON_DONE Status: done ✅ — ${#STOPPED[@]} container(s) stopped"
260	else
261	    warn "Status: ${#FAILED[@]} container(s) could not be stopped"
262	fi
263	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
264
265	[[ ${#FAILED[@]} -gt 0 ]] && exit 1
266	exit 0
267


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Docker Daily Restart =======================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Restarts configured containers every night at 1am as proactive maintenance.
9	#
10	# Called by daily_sync_maintenance.sh via DAILY_MAINTENANCE_SCRIPTS. Runs inside
11	# the daily maintenance window — any service downtime is absorbed by a window
12	# that is already happening. Also drives docker_update.sh in normal mode: the
13	# same DAILY_RESTART_CONTAINERS list is used for both restarts and image pulls,
14	# so there is no second list to maintain.
15	#
16	# ==============================================================================================
17	# DESIGN PRINCIPLES
18	# ==============================================================================================
19	#
20	# Proactive Maintenance
21	#   Daily restarts target containers known to degrade over time without
22	#   crossing a clear failure threshold — connection table growth, scheduler
23	#   state accumulation, session cache bloat. The watchdog cannot detect this
24	#   class of degradation. Scheduled restarts clear it before it becomes visible.
25	#
26	# State Respect
27	#   Running containers are restarted. Stopped containers are left stopped — they
28	#   were intentionally halted and this script has no authority to override that
29	#   decision. This rule is consistent across the entire ecosystem.
30	#
31	# Dependency-Safe Ordering
32	#   Restarts follow the same dependency ordering used by docker_watchdog.sh.
33	#   Services that other containers depend on restart first. A dependent is never
34	#   restarted while its dependency is still coming up.
35	#
36	# ==============================================================================================
37	# OPERATIONAL SAFEGUARDS
38	# ==============================================================================================
39	#
40	# Dependency Ordering
41	#   Containers restart in dependency-safe order using HOST*_WATCHDOG_DEPENDENCIES.
42	#   CONTAINER_DELAY seconds between dependency restart and dependent restart gives
43	#   the dependency time to fully initialise before dependents try to connect.
44	#
45	# Restart Verification
46	#   After each restart, container state is checked after a settle period. A
47	#   container that starts and immediately crashes is marked failed and a
48	#   notification is sent — the script does not silently pass a restart that
49	#   did not stick.
50	#
51	# Timeout Protection
52	#   All docker commands wrapped in a 30 second timeout. A hung Docker daemon
53	#   cannot cause this script to hang indefinitely. Timed-out commands retry
54	#   per RETRY_COUNT before marking as failed.
55	#
56	# Lock Acquisition
57	#   acquire_lock() prevents concurrent execution if a previous run is still active.
58	#
59	# ==============================================================================================
60	# CONFIGURATION
61	# ==============================================================================================
62	#
63	# host*.conf
64	#
65	#   HOST*_DAILY_RESTART_CONTAINERS
66	#       Containers restarted nightly. Also used by docker_update.sh normal mode
67	#       for image pulls — add a container once, it gets both. Aliased by
68	#       detect_hosts() → DAILY_RESTART_CONTAINERS
69	#
70	#   HOST*_WATCHDOG_DEPENDENCIES
71	#       Dependency ordering shared with docker_watchdog.sh. Aliased by
72	#       detect_hosts() → WATCHDOG_DEPENDENCIES
73	#
74	# master.conf
75	#
76	#   RETRY_COUNT
77	#       Retry attempts before giving up on a container
78	#
79	#   SLEEP
80	#       Seconds between retry attempts
81	#
82	#   CONTAINER_DELAY
83	#       Seconds to wait after restarting a dependency before starting its dependents
84	#
85	# ==============================================================================================
86	# RUNTIME MODES
87	# ==============================================================================================
88	#
89	# docker_daily_restart.sh
90	#     Restart all containers in DAILY_RESTART_CONTAINERS
91	#
92	# docker_daily_restart.sh --dry-run
93	#     Preview which containers would be restarted and which would be skipped
94	#
95	# docker_daily_restart.sh --status
96	#     Show configured restart list, container states, and dependency ordering
97	#
98	# docker_daily_restart.sh --log
99	#     Verbose per-container execution output
100	#
101	# ==============================================================================================
102
103	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
104
105	source "$SCRIPT_DIR/../load_config.sh"
106
107	parse_args "$@"
108
109	# ==============================================================================================
110	# ━━━ Setup ━━━
111	# ==============================================================================================
112	if [[ "$EUID" -ne 0 ]]; then
113	    error "Must be run as root"
114	    exit 1
115	fi
116	acquire_lock
117
118	if ! command -v docker &>/dev/null; then
119	    error "Docker command not found — check PATH or Docker installation"
120	    notify "Docker daily restart failed — Docker not found on $(hostname)" "Docker Daily Restart" "warning"
121	    exit 1
122	fi
123
124	# detect_hosts() sets MY_ID and aliases HOST*_DAILY_RESTART_CONTAINERS → DAILY_RESTART_CONTAINERS
125	detect_hosts
126
127	if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
128	    warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to restart"
129	    warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
130	    exit 0
131	fi
132
133	# ==============================================================================================
134	# ━━━ Status ━━━
135	# ==============================================================================================
136	if [[ "$SHOW_STATUS" == true ]]; then
137	    echo ""
138	    echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
139	    echo "$ICON_HOST       Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
140	    echo "$ICON_CONTAINERS Containers: ${DAILY_RESTART_CONTAINERS[*]}"
141	    echo "$ICON_RETRY      Retries:   $RETRY_COUNT"
142	    echo "$ICON_TIME       Sleep:     ${SLEEP}s between retries"
143	    echo "$ICON_NOTIFY     Notify:    unRAID=${NOTIFY_UNRAID:-false} Discord=$([[ -n "${MY_DISCORD_WEBHOOK:-}" ]] && echo enabled || echo disabled)"
144	    echo "$ICON_GEAR       Dry Run:   $DRY_RUN"
145	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
146	    exit 0
147	fi
148
149	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be restarted"
150
151	# ==============================================================================================
152	# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
153	# ==============================================================================================
154
155	# docker_cmd, verify_running, retry_docker — defined in common.sh
156
157	# Builds a dependency-safe restart order from DAILY_RESTART_CONTAINERS.
158	# Containers that are dependencies of others restart first.
159	# Returns ordered list in ORDERED_RESTART array.
160	build_restart_order() {
161	    ORDERED_RESTART=()
162	    local remaining=("${DAILY_RESTART_CONTAINERS[@]}")
163	    local placed=()
164
165	    # First pass — add dependency containers that appear in our list
166	    for container in "${remaining[@]}"; do
167	        [[ -z "$container" ]] && continue
168	        local is_dependency=false
169	        # Check if this container is a dependency of any other in our list
170	        for dep_string in "${WATCHDOG_DEPENDENCIES[@]}"; do
171	            if [[ "$dep_string" == *"$container"* ]]; then
172	                is_dependency=true
173	                break
174	            fi
175	        done
176	        # Also check associative array format
177	        for dependent in "${!WATCHDOG_DEPENDENCIES[@]}"; do
178	            if [[ "${WATCHDOG_DEPENDENCIES[$dependent]}" == *"$container"* ]]; then
179	                is_dependency=true
180	                break
181	            fi
182	        done
183	        if [[ "$is_dependency" == true ]]; then
184	            # Check not already placed
185	            local already=false
186	            for p in "${placed[@]}"; do [[ "$p" == "$container" ]] && already=true && break; done
187	            if [[ "$already" == false ]]; then
188	                ORDERED_RESTART+=("$container")
189	                placed+=("$container")
190	            fi
191	        fi
192	    done
193
194	    # Second pass — add remaining containers (dependents and independents)
195	    for container in "${remaining[@]}"; do
196	        [[ -z "$container" ]] && continue
197	        local already=false
198	        for p in "${placed[@]}"; do [[ "$p" == "$container" ]] && already=true && break; done
199	        if [[ "$already" == false ]]; then
200	            ORDERED_RESTART+=("$container")
201	            placed+=("$container")
202	        fi
203	    done
204
205	}
206
207	# Checks if a container is a dependent of the previously restarted container.
208	# If so, waits CONTAINER_DELAY before restarting to allow dependency to settle.
209	# Usage: check_dependency_delay "$container" "$last_restarted"
210	check_dependency_delay() {
211	    local container="$1"
212	    local last="$2"
213	    [[ -z "$last" ]] && return
214
215	    local deps="${WATCHDOG_DEPENDENCIES[$container]:-}"
216	    if [[ -n "$deps" ]] && [[ "$deps" == *"$last"* ]]; then
217	        echo "  Waiting ${CONTAINER_DELAY}s — $container depends on $last..."
218	        sleep "$CONTAINER_DELAY"
219	    fi
220	}
221
222
223	# ==============================================================================================
224	# ━━━ Daily Restart ━━━
225	# ==============================================================================================
226	echo ""
227	echo "━━━ $ICON_CONTAINERS Daily Restart — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
228	echo "$ICON_HOST $MY_ID ($LOCAL_SERVER_NAME) — ${#DAILY_RESTART_CONTAINERS[@]} container(s)"
229	log "$ICON_CONTAINERS Containers:    ${DAILY_RESTART_CONTAINERS[*]}"
230	log "$ICON_RETRY      Retries:       $RETRY_COUNT"
231	log "$ICON_GEAR       Config:        sleep=${SLEEP}s delay=${CONTAINER_DELAY}s verify-wait=${RESTART_VERIFY_WAIT}s cmd-timeout=${DOCKER_TIMEOUT}s"
232
233	START=$(date +%s)
234	FAILED=()
235	RESTARTED=()
236	SKIPPED=()
237
238	# Build dependency-safe restart order
239	build_restart_order
240	log "$ICON_GEAR Restart order: ${ORDERED_RESTART[*]}"
241
242	LAST_RESTARTED=""
243
244	for container in "${ORDERED_RESTART[@]}"; do
245	    [[ -z "$container" ]] && continue
246	    c_start=$(date +%s)
247	    c_image=$(docker inspect --format '{{.Config.Image}}' "$container" 2>/dev/null || echo "unknown")
248	    log "━━━ $ICON_CONTAINERS $container ($c_image) ━━━"
249
250	    if ! timeout "$DOCKER_TIMEOUT" docker inspect "$container" &>/dev/null; then
251	        warn "$container does not exist — skipping"
252	        continue
253	    fi
254
255	    STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
256
257	    case "$STATUS" in
258	        true)
259	            log "$ICON_RUNNING $container is running — restarting..."
260
261	            # Wait if this container depends on the last one restarted
262	            check_dependency_delay "$container" "$LAST_RESTARTED"
263
264	            if [[ "$DRY_RUN" == true ]]; then
265	                warn "DRY RUN — would restart $container"
266	                RESTARTED+=("$container")
267	            else
268	                if retry_docker docker restart "$container"; then
269	                    # Verify container stayed running after restart
270	                    if verify_running "$container"; then
271	                        log "$ICON_STARTED $container restarted and running in $(format_duration $(( $(date +%s) - c_start ))) ✅"
272	                        RESTARTED+=("$container")
273	                        LAST_RESTARTED="$container"
274	                    else
275	                        error "$container restarted but crashed immediately"
276	                        notify "$container crashed after restart on $(hostname)" "Docker Daily Restart" "warning"
277	                        FAILED+=("$container")
278	                    fi
279	                else
280	                    error "Failed to restart $container after $RETRY_COUNT attempts"
281	                    notify "$container failed to restart on $(hostname)" "Docker Daily Restart" "warning"
282	                    FAILED+=("$container")
283	                fi
284	            fi
285	            ;;
286	        false)
287	            log "$ICON_NOT_RUNNING $container is stopped — skipping"
288	            SKIPPED+=("$container")
289	            ;;
290	        *)
291	            error "Unknown status for $container: $STATUS"
292	            FAILED+=("$container")
293	            ;;
294	    esac
295	done
296
297	END=$(date +%s)
298
299	# ==============================================================================================
300	# ━━━ Prune Old Images ━━━
301	# ==============================================================================================
302	# Restarts above swap containers onto new images — old images are now dangling. Prune immediately.
303	echo ""
304	echo "━━━ $ICON_SYNC Pruning Dangling Images — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
305
306	if [[ "$DRY_RUN" == true ]]; then
307	    warn "DRY RUN — would prune dangling images"
308	    PRUNED_SUMMARY="(dry run)"
309	else
310	    PRUNED_OUTPUT=$(docker image prune -f 2>&1)
311	    [[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/  /'
312	    PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
313	fi
314
315	# ==============================================================================================
316	# ━━━ Summary ━━━
317	# ==============================================================================================
318	echo "━━━━━ $ICON_SUMMARY DAILY RESTART SUMMARY ━━━━━"
319	echo "$ICON_HOST     Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
320	echo "$ICON_TIME     Duration:  $(format_duration $((END - START)))"
321	echo "$ICON_CONTAINERS Scope:   ${#RESTARTED[@]} restarted, ${#SKIPPED[@]} skipped, ${#FAILED[@]} failed"
322	[[ ${#RESTARTED[@]} -gt 0 ]] && log "$ICON_STARTED  Restarted: ${RESTARTED[*]}"
323	[[ ${#SKIPPED[@]} -gt 0 ]]   && log "$ICON_NOT_RUNNING Skipped: ${SKIPPED[*]}"
324	[[ ${#FAILED[@]} -gt 0 ]]    && echo "$ICON_ERROR  Failed:    ${FAILED[*]}"
325	echo "$ICON_SYNC     Pruned:    ${PRUNED_SUMMARY:-none}"
326
327	if [[ "$DRY_RUN" == true ]]; then
328	    warn "DRY RUN — no changes made"
329	elif [[ ${#FAILED[@]} -eq 0 ]]; then
330	    echo "$ICON_DONE     Status:    ALL DONE ✅"
331	    notify "Daily restart complete — ${#RESTARTED[@]} restarted, ${#SKIPPED[@]} skipped on $(hostname)" "Docker Daily Restart" "normal"
332	else
333	    echo "$ICON_ERROR    Status:    ${#FAILED[@]} container(s) failed"
334	    notify "Daily restart completed with errors on $(hostname) — failed: ${FAILED[*]}" "Docker Daily Restart" "warning"
335	fi
336	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
337
338	[[ ${#FAILED[@]} -gt 0 ]] && exit 1
339	exit 0


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Docker Network Connect =====================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Ensures custom Docker networks exist and connects configured containers to
9	# them at every array start.
10	#
11	# Called via ARRAY_START_SCRIPTS — runs early in the array start sequence,
12	# before watchdogs begin their first cycle. Idempotent: safe to re-run at any
13	# time. Silent when everything is already correct. Notifies when a network had
14	# to be created — that only happens after a unRAID update wipes custom networks,
15	# and it is worth knowing when it does.
16	#
17	# ==============================================================================================
18	# OPERATIONAL MODEL
19	# ==============================================================================================
20	#
21	# For each configured network:
22	#
23	#   1. Does the network exist?
24	#        NO  → create it (bridge driver, Docker assigns subnet automatically)
25	#             → send notification — creation is unexpected outside post-update recovery
26	#        YES → skip silently
27	#
28	#   2. For each configured container:
29	#        Already connected  → skip silently
30	#        Not connected      → connect it
31	#        Container missing  → warn and skip — may not be running yet, not fatal
32	#
33	# ==============================================================================================
34	# DESIGN PRINCIPLES
35	# ==============================================================================================
36	#
37	# Idempotent by Design
38	#   Running this script multiple times produces the same result as running it
39	#   once. Skips anything already in the correct state without error or noise.
40	#
41	# Silent When Correct
42	#   Produces no output on a clean run. The absence of output is confirmation
43	#   that everything is already correct.
44	#
45	# Notify on Creation
46	#   Network creation is always notified because it should only happen after a
47	#   unRAID update. If it happens regularly something is misconfigured and the
48	#   operator needs to know.
49	#
50	# ==============================================================================================
51	# OPERATIONAL SAFEGUARDS
52	# ==============================================================================================
53	#
54	# Lock Acquisition
55	#   Prevents concurrent execution via acquire_lock(). Safe to call from
56	#   array start hooks or manually without risk of overlap.
57	#
58	# Host Detection
59	#   detect_hosts() identifies which server is running the script and aliases
60	#   HOST*_NETWORK_CONNECT_* arrays to the correct host's values.
61	#
62	# Docker Daemon Check
63	#   Verifies daemon is responsive before any network operations. Network
64	#   commands against a hung daemon hang indefinitely.
65	#
66	# Timeout Protection
67	#   All docker commands wrapped in timeout. Daemon hangs cannot stall the
68	#   array start sequence.
69	#
70	# Empty Array Guards
71	#   Warns and exits cleanly if NETWORK_CONNECT_NETWORKS or
72	#   NETWORK_CONNECT_CONTAINERS are unconfigured.
73	#
74	# Command Validation
75	#   Validates unRAID notify script before use.
76	#
77	# ==============================================================================================
78	# CONFIGURATION
79	# ==============================================================================================
80	#
81	# host*.conf
82	#
83	#   HOST*_NETWORK_CONNECT_NETWORKS
84	#       Networks to ensure exist at array start. Aliased by detect_hosts() →
85	#       NETWORK_CONNECT_NETWORKS
86	#
87	#   HOST*_NETWORK_CONNECT_CONTAINERS
88	#       Containers to connect to every configured network. Aliased by
89	#       detect_hosts() → NETWORK_CONNECT_CONTAINERS
90	#
91	# ==============================================================================================
92	# RUNTIME MODES
93	# ==============================================================================================
94	#
95	# docker_network_connect.sh
96	#     Ensure all configured networks exist and containers are connected
97	#
98	# docker_network_connect.sh --dry-run
99	#     Preview what would be created or connected without making changes
100	#
101	# docker_network_connect.sh --status
102	#     Show current network state and container connection status
103	#
104	# docker_network_connect.sh --log
105	#     Verbose per-network per-container output
106	#
107	# ==============================================================================================
108
109	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
110
111	source "$SCRIPT_DIR/../load_config.sh"
112
113	parse_args "$@"
114
115	# ==============================================================================================
116	# ━━━ Setup ━━━
117	# ==============================================================================================
118	if [[ "$EUID" -ne 0 ]]; then
119	    error "Must be run as root"
120	    exit 1
121	fi
122
123	acquire_lock
124
125	if ! command -v docker &>/dev/null; then
126	    error "Docker not found — cannot manage networks"
127	    exit 1
128	fi
129
130	# detect_hosts() sets MY_ID and aliases HOST*_NETWORK_CONNECT_* arrays
131	detect_hosts
132
133	# Validate unRAID notify script — used for network creation alerts
134
135	# Docker daemon check — network operations are useless if daemon is hung
136	DOCKER_TIMEOUT=15
137	if ! timeout "$DOCKER_TIMEOUT" docker info >/dev/null 2>&1; then
138	    error "Docker daemon not responding — cannot manage networks"
139	    notify "docker_network_connect failed on $(hostname) — Docker daemon not responding"         "Network Connect" "warning"
140	    exit 1
141	fi
142
143	# Empty array guards
144	if [[ ${#NETWORK_CONNECT_NETWORKS[@]} -eq 0 ]]; then
145	    warn "NETWORK_CONNECT_NETWORKS is empty for $MY_ID — nothing to do"
146	    warn "Check HOST*_NETWORK_CONNECT_NETWORKS in host*.conf"
147	    exit 0
148	fi
149
150	if [[ ${#NETWORK_CONNECT_CONTAINERS[@]} -eq 0 ]]; then
151	    warn "NETWORK_CONNECT_CONTAINERS is empty for $MY_ID — no containers to connect"
152	    warn "Check HOST*_NETWORK_CONNECT_CONTAINERS in host*.conf"
153	    exit 0
154	fi
155
156	# ==============================================================================================
157	# ━━━ Status ━━━
158	# ==============================================================================================
159	if [[ "$SHOW_STATUS" == true ]]; then
160	    echo ""
161	    echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
162	    echo "$ICON_HOST        Identity:   $MY_ID ($LOCAL_SERVER_NAME)"
163	    echo "$ICON_DOCKER_NET  Networks:   ${NETWORK_CONNECT_NETWORKS[*]}"
164	    echo "$ICON_CONTAINERS  Containers: ${NETWORK_CONNECT_CONTAINERS[*]}"
165	    echo ""
166	    echo "━━━ Network State ━━━"
167	    for network in "${NETWORK_CONNECT_NETWORKS[@]}"; do
168	        [[ -z "$network" ]] && continue
169	        if timeout "$DOCKER_TIMEOUT" docker network inspect "$network" &>/dev/null; then
170	            echo "  $ICON_SUCCESS $network — exists"
171	            timeout "$DOCKER_TIMEOUT" docker network inspect "$network" \
172	                --format '  Subnet: {{range .IPAM.Config}}{{.Subnet}}{{end}}' 2>/dev/null
173	        else
174	            echo "  $ICON_ERROR $network — missing (will be created on next run)"
175	        fi
176	    done
177	    echo ""
178	    echo "━━━ Container Connections ━━━"
179	    for container in "${NETWORK_CONNECT_CONTAINERS[@]}"; do
180	        [[ -z "$container" ]] && continue
181	        echo "$ICON_CONTAINERS $container:"
182	        timeout "$DOCKER_TIMEOUT" docker inspect "$container" \
183	            --format '{{range $k, $v := .NetworkSettings.Networks}}    {{$k}}{{"\\n"}}{{end}}' \
184	            2>/dev/null || echo "    not found"
185	    done
186	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
187	    exit 0
188	fi
189
190	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
191
192	# ==============================================================================================
193	# ━━━ Network Ensure + Connect ━━━
194	# ==============================================================================================
195	log "Networks:   ${NETWORK_CONNECT_NETWORKS[*]}"
196	log "Containers: ${NETWORK_CONNECT_CONTAINERS[*]}"
197
198	START=$(date +%s)
199	NETWORKS_CREATED=()
200	CONNECTED=()
201	SKIPPED=()
202	FAILED=()
203
204	for network in "${NETWORK_CONNECT_NETWORKS[@]}"; do
205	    [[ -z "$network" ]] && continue
206
207	    log "Processing network: $network"
208
209	    # ── Step 1 — ensure network exists ───────────────────────────────────────────────────────
210	    if timeout "$DOCKER_TIMEOUT" docker network inspect "$network" &>/dev/null; then
211	        local net_subnet net_driver
212	        net_subnet=$(timeout "$DOCKER_TIMEOUT" docker network inspect "$network" \
213	            --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' 2>/dev/null || echo "unknown")
214	        net_driver=$(timeout "$DOCKER_TIMEOUT" docker network inspect "$network" \
215	            --format '{{.Driver}}' 2>/dev/null || echo "unknown")
216	        log "$ICON_DOCKER_NET $network exists ✅ — driver: $net_driver subnet: $net_subnet"
217	    else
218	        warn "$ICON_DOCKER_NET $network not found — creating (unRAID update may have wiped networks)"
219	        if [[ "$DRY_RUN" == true ]]; then
220	            warn "DRY RUN — would create: $network (bridge, auto subnet)"
221	        else
222	            if timeout "$DOCKER_TIMEOUT" docker network create \
223	                --driver bridge "$network" >/dev/null 2>&1; then
224	                warn "$ICON_DOCKER_NET $network created"
225	                NETWORKS_CREATED+=("$network")
226	                # Network creation is unexpected — notify so user is aware
227	                notify "Docker network created on $(hostname) — $network (unRAID update likely wiped it)" \
228	                    "Network Connect" "normal"
229	            else
230	                error "$network — failed to create"
231	                FAILED+=("$network:create")
232	                continue
233	            fi
234	        fi
235	    fi
236
237	    # ── Step 2 — connect containers to this network ──────────────────────────────────────────
238	    for container in "${NETWORK_CONNECT_CONTAINERS[@]}"; do
239	        [[ -z "$container" ]] && continue
240
241	        # Container not found — warn and skip, not an error
242	        if ! timeout "$DOCKER_TIMEOUT" docker inspect "$container" &>/dev/null; then
243	            warn "$container not found — skipping (may not be running yet)"
244	            continue
245	        fi
246
247	        # Already connected — skip cleanly and silently
248	        if timeout "$DOCKER_TIMEOUT" docker network inspect "$network" \
249	            --format '{{range .Containers}}{{.Name}} {{end}}' 2>/dev/null \
250	            | grep -qw "$container"; then
251	            log "$container already on $network — skipping"
252	            SKIPPED+=("$container→$network")
253	            continue
254	        fi
255
256	        # Connect
257	        if [[ "$DRY_RUN" == true ]]; then
258	            warn "DRY RUN — would connect $container to $network"
259	            continue
260	        fi
261
262	        log "Connecting $container to $network..."
263	        if timeout "$DOCKER_TIMEOUT" docker network connect "$network" "$container" 2>/dev/null; then
264	            log "$container connected to $network"
265	            CONNECTED+=("$container→$network")
266	        else
267	            error "Failed to connect $container to $network"
268	            FAILED+=("$container:$network")
269	        fi
270	    done
271
272	done
273
274	END=$(date +%s)
275
276	# ==============================================================================================
277	# ━━━ Summary ━━━
278	# ==============================================================================================
279	# Always show summary — this runs at array start and output is useful for diagnostics
280	echo ""
281	echo "━━━━━ $ICON_SUMMARY NETWORK CONNECT SUMMARY ━━━━━"
282	echo "$ICON_HOST       Identity:  $MY_ID ($LOCAL_SERVER_NAME)"
283	echo "$ICON_TIME       Duration:  $(format_duration $((END - START)))"
284	[[ ${#NETWORKS_CREATED[@]} -gt 0 ]] && warn "$ICON_DOCKER_NET Created:   ${NETWORKS_CREATED[*]} (networks were missing)"
285	[[ ${#CONNECTED[@]} -gt 0 ]]        && log "Connected: ${CONNECTED[*]}"
286	[[ ${#SKIPPED[@]} -gt 0 ]]          && log "Already connected: ${SKIPPED[*]}"
287	[[ ${#FAILED[@]} -gt 0 ]]           && echo "$ICON_ERROR      Failed:    ${FAILED[*]}"
288
289	if [[ "$DRY_RUN" == true ]]; then
290	    warn "DRY RUN — no changes made"
291	elif [[ ${#FAILED[@]} -gt 0 ]]; then
292	    echo "$ICON_ERROR Status: SOME OPERATIONS FAILED"
293	    notify "Docker network connect failed on $(hostname) — ${FAILED[*]}"         "Network Connect" "warning"
294	elif [[ ${#NETWORKS_CREATED[@]} -gt 0 ]]; then
295	    warn "Networks recreated — ${NETWORKS_CREATED[*]} — unRAID update likely wiped them"
296	else
297	    echo "All networks healthy — no action needed"
298	fi
299	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
300
301	[[ ${#FAILED[@]} -gt 0 ]] && exit 1
302	exit 0


1	#!/bin/bash
2	# ==============================================================================================
3	# ================================= Docker Update ==============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Pulls the latest images for configured containers. Two modes: normal (daily)
9	# and remainder (weekly).
10	#
11	# Normal mode is called by daily_sync_maintenance.sh before docker_daily_restart.sh.
12	# Containers stay running during the pull — no extra downtime beyond what the
13	# nightly restart already causes.
14	#
15	# Remainder mode is called by weekly_sync_maintenance.sh as the final update step.
16	# It catches everything that normal mode and the weekly sync window did not already
17	# update — derived automatically from docker ps, nothing to configure.
18	#
19	# ==============================================================================================
20	# OPERATIONAL MODEL
21	# ==============================================================================================
22	#
23	# Normal mode (daily):
24	#   Targets DAILY_RESTART_CONTAINERS — same list used by docker_daily_restart.sh.
25	#   Pull → compare old vs new image ID → mark updated or already current.
26	#   docker_daily_restart.sh runs after — containers restart onto the fresh image.
27	#
28	# Remainder mode (weekly):
29	#   Targets all currently running containers NOT in:
30	#     DAILY_RESTART_CONTAINERS       — already updated daily
31	#     emby + critical-data profiles  — updated inline by the weekly sync window
32	#     FALLBACK_*_TIER*               — owned by the remote server's update cycle
33	#   Pull → compare → prune dangling images.
34	#
35	# ==============================================================================================
36	# DESIGN PRINCIPLES
37	# ==============================================================================================
38	#
39	# Single List
40	#   Normal mode reuses DAILY_RESTART_CONTAINERS rather than maintaining a
41	#   separate update list. Adding or removing a container from the restart list
42	#   automatically updates the image pull list — one change, both places.
43	#
44	# Version Ownership
45	#   Fallback containers are excluded from remainder mode. This server only runs
46	#   them during a fallback. The remote server owns their version — if remainder
47	#   updates them independently and a handback occurs, the remote's older image
48	#   may not handle data written by the newer version.
49	#
50	# State Respect
51	#   Stopped containers are never targeted. Pulling while stopped adds no value
52	#   and a stopped container was likely halted intentionally.
53	#
54	# ==============================================================================================
55	# OPERATIONAL SAFEGUARDS
56	# ==============================================================================================
57	#
58	# Lock Acquisition
59	#   Prevents concurrent execution via acquire_lock(). Safe to call from
60	#   maintenance scripts without risk of overlap.
61	#
62	# Host Detection
63	#   detect_hosts() identifies which server is running the script and aliases
64	#   HOST*_DAILY_RESTART_CONTAINERS to the correct host's values.
65	#
66	# Root Enforcement
67	#   Docker operations require root privileges.
68	#
69	# DAILY_CONTAINER_UPDATES Toggle
70	#   Normal mode exits cleanly when disabled. docker_daily_restart.sh still runs
71	#   regardless — update and restart are independent operations.
72	#
73	# Fallback Exclusion
74	#   Remainder mode excludes containers owned by the remote server's update cycle
75	#   to prevent version divergence across the fallback boundary.
76	#
77	# Running-Only Filter
78	#   Stopped containers excluded from remainder mode — intentionally down.
79	#
80	# Image ID Comparison
81	#   Containers not restarted unless their image actually changed. Pulls that
82	#   result in "already up to date" produce no restart.
83	#
84	# ==============================================================================================
85	# CONFIGURATION
86	# ==============================================================================================
87	#
88	# master.conf
89	#
90	#   DAILY_CONTAINER_UPDATES
91	#       Enable or disable normal mode. docker_daily_restart.sh runs regardless.
92	#       (default: true)
93	#
94	#   PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data]
95	#       Container names for emby and critical-data profiles — excluded from
96	#       remainder mode (already updated by the weekly sync window)
97	#
98	# host*.conf
99	#
100	#   HOST*_DAILY_RESTART_CONTAINERS
101	#       Containers updated in normal mode. Aliased by detect_hosts() →
102	#       DAILY_RESTART_CONTAINERS
103	#
104	# ==============================================================================================
105	# RUNTIME MODES
106	# ==============================================================================================
107	#
108	# docker_update.sh
109	#     Normal mode — pull latest images for DAILY_RESTART_CONTAINERS
110	#
111	# docker_update.sh --remainder
112	#     Remainder mode — pull all running containers not in managed lists,
113	#     restart those that received updates, prune dangling images
114	#
115	# docker_update.sh --dry-run
116	#     Preview which containers would be pulled without making changes
117	#
118	# docker_update.sh --status
119	#     Show configuration and container list for current mode
120	#
121	# docker_update.sh --log
122	#     Verbose per-container pull and comparison output
123	#
124	# ==============================================================================================
125
126	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
127
128	source "$SCRIPT_DIR/../load_config.sh"
129
130	# Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
131	REMAINDER_MODE=false
132	_filtered_args=()
133	for _arg in "$@"; do
134	    if [[ "$_arg" == "--remainder" ]]; then
135	        REMAINDER_MODE=true
136	    else
137	        _filtered_args+=("$_arg")
138	    fi
139	done
140	unset _arg
141
142	parse_args "${_filtered_args[@]}"
143	unset _filtered_args
144
145	# ==============================================================================================
146	# ━━━ Setup ━━━
147	# ==============================================================================================
148	if [[ "$EUID" -ne 0 ]]; then
149	    error "Must be run as root"
150	    exit 1
151	fi
152
153	acquire_lock
154
155	if ! command -v docker &>/dev/null; then
156	    error "Docker command not found"
157	    exit 1
158	fi
159
160	detect_hosts
161
162	# ==============================================================================================
163	# ━━━ Container Discovery ━━━
164	# ==============================================================================================
165	if [[ "$REMAINDER_MODE" == true ]]; then
166	    declare -A _exclude=()
167
168	    # Daily containers — updated by docker_update.sh normal mode
169	    for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
170	        [[ -n "$_c" ]] && _exclude["$_c"]=1
171	    done
172
173	    # Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
174	    _weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
175	    read -r -a _weekly_arr <<< "$_weekly_str"
176	    for _c in "${_weekly_arr[@]}"; do
177	        [[ -n "$_c" ]] && _exclude["$_c"]=1
178	    done
179	    unset _weekly_str _weekly_arr
180
181	    # Fallback coverage containers — owned by the remote server's update cycle.
182	    # This server runs them during fallback but should never update them independently.
183	    # Updating them here risks version divergence: if remote's writeback after handback
184	    # encounters data written by a newer version, it may not handle it correctly.
185	    for _tier in 1 2 3 4; do
186	        _tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
187	        eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
188	        for _c in "${_tier_arr[@]}"; do
189	            [[ -n "$_c" ]] && _exclude["$_c"]=1
190	        done
191	    done
192	    unset _tier _tier_var _tier_arr _c
193
194	    mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
195	    TARGET_CONTAINERS=()
196	    for _c in "${_all_running[@]}"; do
197	        [[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
198	    done
199	    unset _all_running _exclude _c
200	else
201	    if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
202	        echo "DAILY_CONTAINER_UPDATES=false — skipping container updates"
203	        exit 0
204	    fi
205
206	    if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
207	        warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
208	        warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
209	        exit 0
210	    fi
211
212	    TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
213	fi
214
215	# ==============================================================================================
216	# ━━━ Status ━━━
217	# ==============================================================================================
218	if [[ "$SHOW_STATUS" == true ]]; then
219	    echo ""
220	    echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
221	    echo "$ICON_HOST       Identity:   $MY_ID ($LOCAL_SERVER_NAME)"
222	    echo "$ICON_GEAR       Mode:       $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
223	    if [[ "$REMAINDER_MODE" == true ]]; then
224	        echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
225	        for _c in "${TARGET_CONTAINERS[@]}"; do echo "  $_c"; done
226	    else
227	        echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
228	        echo "$ICON_GEAR       Enabled:    ${DAILY_CONTAINER_UPDATES:-true}"
229	    fi
230	    echo "$ICON_GEAR       Dry Run:    $DRY_RUN"
231	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
232	    exit 0
233	fi
234
235	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
236
237	if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
238	    echo "No containers to update"
239	    exit 0
240	fi
241
242	# ==============================================================================================
243	# ━━━ Pull Updates ━━━
244	# ==============================================================================================
245	echo ""
246	if [[ "$REMAINDER_MODE" == true ]]; then
247	    echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
248	    echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
249	    log "$ICON_CONTAINERS Remainder targets: ${TARGET_CONTAINERS[*]}"
250	else
251	    echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
252	    log "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
253	fi
254	echo ""
255
256	START=$(date +%s)
257	UPDATED=()
258	UP_TO_DATE=()
259	FAILED=()
260	OLD_IMAGE_IDS=()   # old image IDs to explicitly remove after rebuilds
261	SKIPPED=()
262
263	for container in "${TARGET_CONTAINERS[@]}"; do
264	    [[ -z "$container" ]] && continue
265	    log "━━━ $ICON_CONTAINERS $container ━━━"
266
267	    if ! docker inspect "$container" &>/dev/null; then
268	        warn "$container — not found, skipping"
269	        SKIPPED+=("$container")
270	        continue
271	    fi
272
273	    IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
274	    if [[ -z "$IMAGE" ]]; then
275	        warn "$container — could not determine image, skipping"
276	        SKIPPED+=("$container")
277	        continue
278	    fi
279
280	    log "$container — image: $IMAGE"
281
282	    if [[ "$DRY_RUN" == true ]]; then
283	        warn "DRY RUN — would pull: $IMAGE"
284	        UPDATED+=("$container")
285	        continue
286	    fi
287
288	    # Capture the image ID the container is currently running on, and the
289	    # image ID :latest points to before the pull. After pulling, we rebuild if
290	    # either a new digest landed OR the container is behind what :latest is now.
291	    CONTAINER_IMAGE_ID=$(docker inspect "$container" --format='{{.Image}}' 2>/dev/null || echo "")
292	    OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
293
294	    log "$ICON_SYNC  Pulling $IMAGE..."
295	    if [[ "$ENABLE_LOGGING" == "true" ]]; then
296	        docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/  /'
297	        _pull_rc=${PIPESTATUS[0]}
298	    else
299	        docker pull "$IMAGE" >/dev/null 2>&1
300	        _pull_rc=$?
301	    fi
302	    NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
303
304	    if [[ $_pull_rc -eq 0 ]]; then
305	        _pull_new=$(  [[ -n "$OLD_ID" && "$OLD_ID" != "$NEW_ID" ]] && echo true || echo false)
306	        _container_behind=$([[ -n "$CONTAINER_IMAGE_ID" && -n "$NEW_ID" && "$CONTAINER_IMAGE_ID" != "$NEW_ID" ]] && echo true || echo false)
307
308	        if [[ "$_pull_new" == true || "$_container_behind" == true ]]; then
309	            [[ "$_pull_new" == true ]]       && log "$ICON_DONE  $container — new image (${OLD_ID:7:12} → ${NEW_ID:7:12})"
310	            [[ "$_container_behind" == true && "$_pull_new" == false ]] && log "$ICON_DONE  $container — image already pulled, container behind (${CONTAINER_IMAGE_ID:7:12} → ${NEW_ID:7:12})"
311	            UPDATED+=("$container")
312	            OLD_IMAGE_IDS+=("$CONTAINER_IMAGE_ID")
313	        else
314	            log "$container — up to date (${NEW_ID:7:12})"
315	            UP_TO_DATE+=("$container")
316	        fi
317	    else
318	        warn "$container — pull failed ($IMAGE)"
319	        FAILED+=("$container")
320	    fi
321
322	done
323
324	# ── Recreate containers that received a new image ────────────────────────────
325	# docker restart uses the image ID baked in at creation time — it never picks
326	# up the new digest. rebuild_container reads the stored XML template, stops the
327	# old container, recreates it (new image, same config), then prunes the old image.
328	REBUILT=()
329	REBUILD_FAILED=()
330	if [[ ${#UPDATED[@]} -gt 0 ]]; then
331	    for container in "${UPDATED[@]}"; do
332	        [[ -z "$container" ]] && continue
333	        if [[ "$DRY_RUN" == true ]]; then
334	            warn "DRY RUN — would rebuild $container from template"
335	            REBUILT+=("$container")
336	            continue
337	        fi
338	        log "$ICON_SYNC Rebuilding $container from template on new image..."
339	        if platform_rebuild_container "$container"; then
340	            log "$ICON_DONE $container rebuilt ✅"
341	            REBUILT+=("$container")
342	        else
343	            error "Failed to rebuild $container — will be picked up by docker_daily_restart.sh"
344	            notify "$container failed to rebuild after image update on $(hostname)" "Docker Update" "warning"
345	            REBUILD_FAILED+=("$container")
346	        fi
347	    done
348	fi
349
350	# ── Remove old images ────────────────────────────────────────────────────────
351	# Explicitly rmi by the IDs captured before each pull. Tagged images are never
352	# caught by dangling-only prune, so this is the only reliable cleanup path.
353	# Fall through to dangling prune to catch any leftovers from other update paths.
354	if [[ "$DRY_RUN" == true ]]; then
355	    warn "DRY RUN — would remove ${#OLD_IMAGE_IDS[@]} old image(s) and prune dangling"
356	    PRUNED_SUMMARY="(dry run)"
357	else
358	    for _old_id in "${OLD_IMAGE_IDS[@]}"; do
359	        docker rmi "$_old_id" >/dev/null 2>&1 || true
360	    done
361	    PRUNED_OUTPUT=$(docker image prune -f 2>&1)
362	    [[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/  /'
363	    PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
364	fi
365
366	END=$(date +%s)
367
368	# ==============================================================================================
369	# ━━━ Summary ━━━
370	# ==============================================================================================
371	if [[ "$REMAINDER_MODE" == true ]]; then
372	    echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
373	else
374	    echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
375	fi
376	echo "$ICON_HOST       Identity:    $MY_ID ($LOCAL_SERVER_NAME)"
377	echo "$ICON_TIME       Duration:    $(format_duration $(( END - START )))"
378	if [[ ${#UPDATED[@]} -gt 0 ]]; then
379	    echo "$ICON_DONE       Updated:     ${#UPDATED[@]}"
380	    log "  ${UPDATED[*]}"
381	fi
382	[[ ${#REBUILT[@]} -gt 0 ]]       && echo "$ICON_SYNC       Rebuilt:     ${#REBUILT[@]}"
383	[[ ${#REBUILD_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR      Rebuild fail:${REBUILD_FAILED[*]}"
384	[[ ${#UP_TO_DATE[@]} -gt 0 ]]    && log "$ICON_RUNNING    Up to date:  ${UP_TO_DATE[*]}"
385	[[ ${#SKIPPED[@]} -gt 0 ]]       && log "$ICON_WARN       Skipped:     ${SKIPPED[*]}"
386	[[ ${#FAILED[@]} -gt 0 ]]        && echo "$ICON_ERROR      Failed:      ${FAILED[*]}"
387	echo "$ICON_SYNC       Pruned:      ${PRUNED_SUMMARY:-none}"
388
389	if [[ "$DRY_RUN" == true ]]; then
390	    warn "DRY RUN — no images pulled"
391	elif [[ ${#FAILED[@]} -eq 0 ]]; then
392	    echo "$ICON_DONE Status: done ✅ — ${#UPDATED[@]} updated, ${#UP_TO_DATE[@]} current"
393	else
394	    warn "Status: ${#FAILED[@]} pull(s) failed — restart will proceed with existing images"
395	fi
396	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
397
398	# Pull failures are non-fatal — restart proceeds regardless
399	exit 0
400


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Docker Update — Remaining ======================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Weekly sweep that pulls the latest image for every running container not
9	# already covered by the daily or weekly managed update cycles. Restarts
10	# containers that received a new image, then prunes dangling images.
11	#
12	# Called by weekly_sync_maintenance.sh as the final step in the weekly window.
13	# Derives its target list automatically from docker ps minus the two managed
14	# lists — there is nothing to configure for this script.
15	#
16	# Together with docker_update.sh (normal + remainder modes), every deployed
17	# container receives at least one image pull per week without any per-container
18	# configuration required here.
19	#
20	# ==============================================================================================
21	# OPERATIONAL SAFEGUARDS
22	# ==============================================================================================
23	#
24	# Lock Acquisition
25	#   Prevents concurrent execution via acquire_lock(). Safe to call from
26	#   weekly maintenance scripts without risk of overlap.
27	#
28	# Host Detection
29	#   detect_hosts() identifies which server is running the script and aliases
30	#   HOST*_DAILY_RESTART_CONTAINERS and HOST*_WEEKLY_RESTART_CONTAINERS to
31	#   the correct host's values for exclusion.
32	#
33	# Root Enforcement
34	#   Docker operations require root privileges.
35	#
36	# WEEKLY_REMAINING_UPDATES Toggle
37	#   Exits cleanly when disabled via master.conf.
38	#
39	# Running-Only Filter
40	#   Stopped containers excluded — intentionally down, pulling adds no value.
41	#
42	# Image ID Comparison
43	#   Containers not restarted unless their image actually changed.
44	#
45	# ==============================================================================================
46	# CONFIGURATION
47	# ==============================================================================================
48	#
49	# master.conf
50	#
51	#   WEEKLY_REMAINING_UPDATES
52	#       Enable or disable this script. (default: true)
53	#       To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
54	#
55	# host*.conf
56	#
57	#   HOST*_DAILY_RESTART_CONTAINERS
58	#       Excluded from this script — already updated daily. Aliased by
59	#       detect_hosts() → DAILY_RESTART_CONTAINERS
60	#
61	#   HOST*_WEEKLY_RESTART_CONTAINERS
62	#       Excluded from this script — already updated by weekly sync window.
63	#       Aliased by detect_hosts() → WEEKLY_RESTART_CONTAINERS
64	#
65	# ==============================================================================================
66	# RUNTIME MODES
67	# ==============================================================================================
68	#
69	# docker_update_remaining.sh
70	#     Pull all remaining running containers, restart those updated, prune images
71	#
72	# docker_update_remaining.sh --dry-run
73	#     Preview which containers would be pulled and restarted
74	#
75	# docker_update_remaining.sh --status
76	#     Show exclusion lists and current remaining container count
77	#
78	# docker_update_remaining.sh --log
79	#     Verbose per-container pull and restart output
80	#
81	# ==============================================================================================
82
83	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
84
85	source "$SCRIPT_DIR/../load_config.sh"
86
87	parse_args "$@"
88
89	# ==============================================================================================
90	# ━━━ Setup ━━━
91	# ==============================================================================================
92	if [[ "$EUID" -ne 0 ]]; then
93	    error "Must be run as root"
94	    exit 1
95	fi
96
97	acquire_lock
98
99	if ! command -v docker &>/dev/null; then
100	    error "Docker command not found"
101	    exit 1
102	fi
103
104	detect_hosts
105
106	if [[ "${WEEKLY_REMAINING_UPDATES:-true}" != "true" ]]; then
107	    echo "WEEKLY_REMAINING_UPDATES=false — skipping remaining container updates"
108	    exit 0
109	fi
110
111	# ── Build exclusion set from daily + weekly managed lists ─────────────────────────────────────
112	declare -A EXCLUDED
113	for c in "${DAILY_RESTART_CONTAINERS[@]}"  "${WEEKLY_RESTART_CONTAINERS[@]}"; do
114	    [[ -n "$c" ]] && EXCLUDED["$c"]=1
115	done
116
117	# ── Get all running containers ────────────────────────────────────────────────────────────────
118	mapfile -t ALL_RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
119
120	# ── Derive remainder: running minus excluded ──────────────────────────────────────────────────
121	REMAINING=()
122	for c in "${ALL_RUNNING[@]}"; do
123	    [[ -z "$c" ]] && continue
124	    [[ -n "${EXCLUDED[$c]:-}" ]] && continue
125	    REMAINING+=("$c")
126	done
127
128	# ==============================================================================================
129	# ━━━ Status ━━━
130	# ==============================================================================================
131	if [[ "$SHOW_STATUS" == true ]]; then
132	    echo ""
133	    echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
134	    echo "$ICON_HOST       Identity:    $MY_ID ($LOCAL_SERVER_NAME)"
135	    echo "$ICON_GEAR       Enabled:     ${WEEKLY_REMAINING_UPDATES:-true}"
136	    echo "$ICON_CONTAINERS All running: ${#ALL_RUNNING[@]}"
137	    echo "$ICON_CONTAINERS Excluded:    ${!EXCLUDED[*]}"
138	    echo "$ICON_CONTAINERS Remaining:   ${REMAINING[*]:-none}"
139	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
140	    exit 0
141	fi
142
143	if [[ ${#REMAINING[@]} -eq 0 ]]; then
144	    echo "No remaining containers to update — all running containers are covered by daily/weekly lists"
145	    exit 0
146	fi
147
148	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled or containers restarted"
149
150	# ==============================================================================================
151	# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
152	# ==============================================================================================
153
154	# docker_cmd, retry_docker, verify_running — defined in common.sh
155
156	# ==============================================================================================
157	# ━━━ Pull Updates ━━━
158	# ==============================================================================================
159	echo ""
160	echo "━━━ $ICON_CONTAINERS Docker Update (Remaining) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
161	log "$ICON_CONTAINERS Containers: ${REMAINING[*]}"
162	log "$ICON_CONTAINERS Excluded (managed elsewhere): ${!EXCLUDED[*]}"
163	echo ""
164
165	START=$(date +%s)
166	UPDATED=()
167	UP_TO_DATE=()
168	FAILED=()
169	OLD_IMAGE_IDS=()
170
171	for container in "${REMAINING[@]}"; do
172	    [[ -z "$container" ]] && continue
173	    log "━━━ $ICON_CONTAINERS $container ━━━"
174
175	    IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
176	    if [[ -z "$IMAGE" ]]; then
177	        warn "$container — could not determine image, skipping"
178	        FAILED+=("$container")
179	        continue
180	    fi
181
182	    log "$container — image: $IMAGE"
183
184	    if [[ "$DRY_RUN" == true ]]; then
185	        warn "DRY RUN — would pull: $IMAGE"
186	        UPDATED+=("$container")
187	        continue
188	    fi
189
190	    CONTAINER_IMAGE_ID=$(docker inspect "$container" --format='{{.Image}}' 2>/dev/null || echo "")
191	    OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
192
193	    log "$ICON_SYNC  Pulling $IMAGE..."
194	    if [[ "$ENABLE_LOGGING" == "true" ]]; then
195	        docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/  /'
196	        _pull_rc=${PIPESTATUS[0]}
197	    else
198	        docker pull "$IMAGE" >/dev/null 2>&1
199	        _pull_rc=$?
200	    fi
201	    NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
202
203	    if [[ $_pull_rc -eq 0 ]]; then
204	        _pull_new=$(  [[ -n "$OLD_ID" && "$OLD_ID" != "$NEW_ID" ]] && echo true || echo false)
205	        _container_behind=$([[ -n "$CONTAINER_IMAGE_ID" && -n "$NEW_ID" && "$CONTAINER_IMAGE_ID" != "$NEW_ID" ]] && echo true || echo false)
206
207	        if [[ "$_pull_new" == true || "$_container_behind" == true ]]; then
208	            [[ "$_pull_new" == true ]]       && log "$ICON_DONE  $container — new image (${OLD_ID:7:12} → ${NEW_ID:7:12})"
209	            [[ "$_container_behind" == true && "$_pull_new" == false ]] && log "$ICON_DONE  $container — image already pulled, container behind (${CONTAINER_IMAGE_ID:7:12} → ${NEW_ID:7:12})"
210	            UPDATED+=("$container")
211	            OLD_IMAGE_IDS+=("$CONTAINER_IMAGE_ID")
212	        else
213	            log "$container — up to date (${NEW_ID:7:12})"
214	            UP_TO_DATE+=("$container")
215	        fi
216	    else
217	        warn "$container — pull failed ($IMAGE)"
218	        FAILED+=("$container")
219	    fi
220
221	done
222
223	# ==============================================================================================
224	# ━━━ Restart Updated Containers ━━━
225	# ==============================================================================================
226	RESTARTED=()
227	RESTART_FAILED=()
228	SKIPPED_STOPPED=()
229
230	if [[ ${#UPDATED[@]} -gt 0 ]]; then
231	    echo ""
232	    echo "━━━ $ICON_CONTAINERS Restarting Updated Containers — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
233	    log "$ICON_CONTAINERS Containers with new image: ${UPDATED[*]}"
234
235	    for container in "${UPDATED[@]}"; do
236	        [[ -z "$container" ]] && continue
237	        log "━━━ $ICON_CONTAINERS $container ━━━"
238
239	        STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
240
241	        if [[ "$STATUS" != "true" ]]; then
242	            log "$ICON_NOT_RUNNING $container is stopped — skipping restart (respecting stopped state)"
243	            SKIPPED_STOPPED+=("$container")
244	            continue
245	        fi
246
247	        if [[ "$DRY_RUN" == true ]]; then
248	            warn "DRY RUN — would restart $container"
249	            RESTARTED+=("$container")
250	            continue
251	        fi
252
253	        log "$ICON_RUNNING $container — recreating from template on new image..."
254	        if platform_rebuild_container "$container"; then
255	            if verify_running "$container"; then
256	                log "$ICON_DONE  $container recreated and running ✅"
257	                RESTARTED+=("$container")
258	            else
259	                error "$container recreated but not running — may be intentionally stopped"
260	                RESTARTED+=("$container")
261	            fi
262	        else
263	            error "Failed to rebuild $container from template"
264	            notify "$container failed to rebuild after update on $(hostname)" "Docker Update Remaining" "warning"
265	            RESTART_FAILED+=("$container")
266	        fi
267	    done
268	else
269	    log "No containers received a new image — nothing to restart"
270	fi
271
272	# ==============================================================================================
273	# ━━━ Prune Old Images ━━━
274	# ==============================================================================================
275	echo ""
276	echo "━━━ $ICON_SYNC Pruning Dangling Images — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
277
278	if [[ "$DRY_RUN" == true ]]; then
279	    warn "DRY RUN — would remove ${#OLD_IMAGE_IDS[@]} old image(s) and prune dangling"
280	    PRUNED_SUMMARY="(dry run)"
281	else
282	    for _old_id in "${OLD_IMAGE_IDS[@]}"; do
283	        docker rmi "$_old_id" >/dev/null 2>&1 || true
284	    done
285	    PRUNED_OUTPUT=$(docker image prune -f 2>&1)
286	    [[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/  /'
287	    PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
288	fi
289
290	END=$(date +%s)
291
292	# ==============================================================================================
293	# ━━━ Summary ━━━
294	# ==============================================================================================
295	echo ""
296	echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINING) SUMMARY ━━━━━"
297	echo "$ICON_HOST       Identity:    $MY_ID ($LOCAL_SERVER_NAME)"
298	echo "$ICON_TIME       Duration:    $(format_duration $(( END - START )))"
299	echo "$ICON_CONTAINERS Scope:       ${#ALL_RUNNING[@]} running — ${#EXCLUDED[@]} managed = ${#REMAINING[@]} checked"
300	if [[ ${#UPDATED[@]} -gt 0 ]]; then
301	    echo "$ICON_DONE       New image:   ${#UPDATED[@]}"
302	    log "  ${UPDATED[*]}"
303	fi
304	[[ ${#UP_TO_DATE[@]} -gt 0 ]]    && log "$ICON_RUNNING    Up to date:  ${UP_TO_DATE[*]}"
305	[[ ${#FAILED[@]} -gt 0 ]]        && echo "$ICON_ERROR      Pull failed: ${FAILED[*]}"
306	if [[ ${#RESTARTED[@]} -gt 0 ]]; then
307	    echo "$ICON_DONE       Restarted:   ${#RESTARTED[@]}"
308	    log "  ${RESTARTED[*]}"
309	fi
310	[[ ${#SKIPPED_STOPPED[@]} -gt 0 ]] && log "$ICON_WARN     Not running: ${SKIPPED_STOPPED[*]} (skipped restart)"
311	[[ ${#RESTART_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR     Restart fail:${RESTART_FAILED[*]}"
312	echo "$ICON_SYNC       Pruned:      ${PRUNED_SUMMARY:-none}"
313
314	ALL_FAILED=$(( ${#FAILED[@]} + ${#RESTART_FAILED[@]} ))
315	if [[ "$DRY_RUN" == true ]]; then
316	    warn "DRY RUN — no changes made"
317	elif [[ "$ALL_FAILED" -eq 0 ]]; then
318	    echo "$ICON_DONE Status: done ✅ — ${#RESTARTED[@]} restarted, ${#UP_TO_DATE[@]} current"
319	else
320	    warn "Status: $ALL_FAILED error(s) — ${#FAILED[@]} pull failure(s), ${#RESTART_FAILED[@]} restart failure(s)"
321	fi
322	echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
323
324	[[ "$ALL_FAILED" -gt 0 ]] && exit 1
325	exit 0
326