feat: Tailscale resolution hardening, partnership offboard completion, Emby provisioning

common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
  parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
  is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
  PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)

Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
  own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
  problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
  if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
  matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
  or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
  for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
  cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
  dry runs)

master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)

master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS

Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
2026-05-10 20:09:13 -04:00
co-authored by Claude Sonnet 4.6
parent 6948755c86
commit 0ae31b5fa6
33 changed files with 1241 additions and 155 deletions
+178
View File
@@ -0,0 +1,178 @@
#!/bin/bash
# ==============================================================================================
# ============================= Docker Container Stop ==========================================
# ==============================================================================================
# Stops all running Docker containers one at a time, verifying each is stopped before
# moving to the next. Called by array_stop.sh as part of a planned shutdown sequence.
#
# ── STOP SEQUENCE PER CONTAINER ──────────────────────────────────────────────────────────────
# 1. docker stop -t 30 (SIGTERM + 30s grace period — docker sends SIGKILL if needed)
# 2. Verify stopped — if still running, retry up to RETRY_COUNT times
# 3. docker kill (SIGKILL) if all retries exhausted
# 4. Final verify — error if still running after force-kill
# Never moves to the next container until the current one is confirmed stopped.
#
# ── WHY SEQUENTIAL ────────────────────────────────────────────────────────────────────────────
# Containers may have dependencies — stopping one at a time avoids abruptly severing a
# service while its dependents are still running and trying to use it.
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# Root check — docker requires root
# Per-container verify — confirmed stopped before proceeding to next
# Retry loop — RETRY_COUNT attempts before escalating to force-kill
# SIGTERM → SIGKILL — graceful then forced, never skips graceful
# DOCKER_TIMEOUT — all docker calls protected against hung daemon
# notify on failures — alert if any container cannot be stopped
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# RETRY_COUNT — retry attempts before force-kill (default 3)
# SLEEP — seconds between retries
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_container_stop.sh — stop all running containers
# docker_container_stop.sh --dry-run — show which containers would be stopped
# docker_container_stop.sh --status — show running containers and exit
# docker_container_stop.sh --log — verbose output
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
if ! command -v docker &>/dev/null; then
error "Docker command not found"
exit 1
fi
detect_hosts
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no containers will be stopped"
DOCKER_TIMEOUT=30
DOCKER_STOP_TIMEOUT=30 # grace period for SIGTERM before docker sends SIGKILL internally
_RETRY_COUNT="${RETRY_COUNT:-3}"
# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$SHOW_STATUS" == true ]]; then
mapfile -t RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
echo ""
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_CONTAINERS Running: ${#RUNNING[@]} container(s)"
echo "$ICON_RETRY Retries: $_RETRY_COUNT"
for c in "${RUNNING[@]}"; do echo " $ICON_RUNNING $c"; done
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# ==============================================================================================
# ━━━ Stop Containers ━━━
# ==============================================================================================
mapfile -t RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
echo ""
echo "━━━ $ICON_CONTAINERS Docker Container Stop — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
if [[ ${#RUNNING[@]} -eq 0 ]]; then
log "No running containers — nothing to do"
exit 0
fi
echo "$ICON_CONTAINERS Containers: ${#RUNNING[@]} running"
echo ""
START=$(date +%s)
STOPPED=()
FAILED=()
for container in "${RUNNING[@]}"; do
[[ -z "$container" ]] && continue
echo "━━━ $ICON_CONTAINERS $container ━━━"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would stop $container"
STOPPED+=("$container")
echo ""
continue
fi
attempt=1
success=false
while [[ "$attempt" -le "$_RETRY_COUNT" ]]; do
log "$ICON_RETRY Attempt $attempt of $_RETRY_COUNT — stopping $container..."
timeout "$DOCKER_TIMEOUT" docker stop -t "$DOCKER_STOP_TIMEOUT" "$container" \
>/dev/null 2>&1
STATE=$(timeout "$DOCKER_TIMEOUT" docker inspect \
-f '{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
if [[ "$STATE" != "true" ]]; then
echo "$ICON_DONE $container stopped ✅"
STOPPED+=("$container")
success=true
break
fi
warn "Attempt $attempt failed — $container still running"
(( attempt++ ))
[[ "$attempt" -le "$_RETRY_COUNT" ]] && sleep "${SLEEP:-5}"
done
if [[ "$success" == false ]]; then
warn "$container — retries exhausted, force-killing with SIGKILL..."
timeout "$DOCKER_TIMEOUT" docker kill "$container" >/dev/null 2>&1 || true
STATE=$(timeout "$DOCKER_TIMEOUT" docker inspect \
-f '{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
if [[ "$STATE" != "true" ]]; then
warn "$container force-killed ✅"
STOPPED+=("$container")
else
error "$container still running after SIGKILL — manual intervention needed"
notify "$container could not be stopped on $(hostname) ($MY_ID)" \
"Docker Container Stop" "warning"
FAILED+=("$container")
fi
fi
echo ""
done
END=$(date +%s)
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
echo "━━━━━ $ICON_SUMMARY DOCKER CONTAINER STOP SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo "$ICON_CONTAINERS Scope: ${#RUNNING[@]} running → ${#STOPPED[@]} stopped"
[[ ${#STOPPED[@]} -gt 0 ]] && echo "$ICON_DONE Stopped: ${STOPPED[*]}"
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${FAILED[*]}"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no containers stopped"
elif [[ ${#FAILED[@]} -eq 0 ]]; then
log "$ICON_DONE Status: done ✅ — ${#STOPPED[@]} container(s) stopped"
else
warn "Status: ${#FAILED[@]} container(s) could not be stopped"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
[[ ${#FAILED[@]} -gt 0 ]] && exit 1
exit 0
-26
View File
@@ -107,32 +107,6 @@ fi
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Retries a docker command up to RETRY_COUNT times with SLEEP seconds between attempts.
# Usage: retry_docker docker restart ContainerName
retry_docker() {
local attempt=1
while [[ "$attempt" -le "$RETRY_COUNT" ]]; do
log "$ICON_RETRY Attempt $attempt of $RETRY_COUNT: $*"
if "$@"; then
log "Succeeded on attempt $attempt"
return 0
else
warn "Attempt $attempt failed"
(( attempt++ ))
[[ "$attempt" -le "$RETRY_COUNT" ]] && sleep "$SLEEP"
fi
done
error "Command failed after $RETRY_COUNT attempts: $*"
return 1
}
# ==============================================================================================
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Wraps docker commands with a 30 second timeout.
# Prevents a hung Docker daemon from causing the script to hang indefinitely.
# Usage: docker_cmd docker restart ContainerName
View File
+134 -11
View File
@@ -3,7 +3,8 @@
# ============================= Docker Update — Remaining ======================================
# ==============================================================================================
# Pulls the latest image for every running container NOT already covered by the daily or
# weekly update/restart cycles. Runs at the end of the weekly maintenance window.
# weekly update/restart cycles. Restarts containers that received a new image, then prunes
# dangling images. Runs at the end of the weekly maintenance window.
#
# ── WHAT THIS COVERS ──────────────────────────────────────────────────────────────────────────
# Daily update: DAILY_RESTART_CONTAINERS — auth stack, NPM, Dispatcharr, etc.
@@ -14,6 +15,12 @@
# pull per week, with no container list to maintain here — it derives the remainder
# automatically from `docker ps` minus the two managed lists.
#
# ── WHAT THIS DOES ────────────────────────────────────────────────────────────────────────────
# 1. Pull latest image for each remaining running container
# 2. Restart containers whose image ID changed (new update landed)
# 3. Prune dangling images left behind by the updates
# Containers already up to date are not restarted.
#
# ── EXCLUSION LOGIC ───────────────────────────────────────────────────────────────────────────
# Exclusion set = DAILY_RESTART_CONTAINERS + WEEKLY_RESTART_CONTAINERS (aliased by detect_hosts)
# Only running containers are targeted — stopped containers are intentionally excluded
@@ -29,7 +36,7 @@
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# docker_update_remaining.sh — normal run
# docker_update_remaining.sh --dry-run — show which containers would be pulled
# docker_update_remaining.sh --dry-run — show which containers would be pulled/restarted
# docker_update_remaining.sh --log — verbose output
# docker_update_remaining.sh --status — show config and exit
# ==============================================================================================
@@ -97,7 +104,48 @@ if [[ ${#REMAINING[@]} -eq 0 ]]; then
exit 0
fi
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled or containers restarted"
# ==============================================================================================
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
DOCKER_TIMEOUT=30
docker_cmd() {
timeout "$DOCKER_TIMEOUT" "$@"
local exit_code=$?
if [[ "$exit_code" -eq 124 ]]; then
error "Docker command timed out after ${DOCKER_TIMEOUT}s: $*"
return 1
fi
return "$exit_code"
}
retry_docker() {
local attempt=1
while [[ "$attempt" -le "$RETRY_COUNT" ]]; do
log "$ICON_RETRY Attempt $attempt of $RETRY_COUNT: $*"
if docker_cmd "$@"; then
log "Succeeded on attempt $attempt"
return 0
else
warn "Attempt $attempt failed"
(( attempt++ ))
[[ "$attempt" -le "$RETRY_COUNT" ]] && sleep "$SLEEP"
fi
done
error "Command failed after $RETRY_COUNT attempts: $*"
return 1
}
RESTART_VERIFY_WAIT=5
verify_running() {
local container="$1"
sleep "$RESTART_VERIFY_WAIT"
local state
state=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
[[ "$state" == "true" ]]
}
# ==============================================================================================
# ━━━ Pull Updates ━━━
@@ -155,27 +203,102 @@ for container in "${REMAINING[@]}"; do
echo ""
done
# ==============================================================================================
# ━━━ Restart Updated Containers ━━━
# ==============================================================================================
RESTARTED=()
RESTART_FAILED=()
SKIPPED_STOPPED=()
if [[ ${#UPDATED[@]} -gt 0 ]]; then
echo ""
echo "━━━ $ICON_CONTAINERS Restarting Updated Containers — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo "$ICON_CONTAINERS Containers with new image: ${UPDATED[*]}"
echo ""
for container in "${UPDATED[@]}"; do
[[ -z "$container" ]] && continue
echo "━━━ $ICON_CONTAINERS $container ━━━"
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
if [[ "$STATUS" != "true" ]]; then
echo "$ICON_NOT_RUNNING $container is stopped — skipping restart (respecting stopped state)"
SKIPPED_STOPPED+=("$container")
echo ""
continue
fi
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart $container"
RESTARTED+=("$container")
echo ""
continue
fi
echo "$ICON_RUNNING $container is running — restarting on new image..."
if retry_docker docker restart "$container"; then
if verify_running "$container"; then
echo "$ICON_DONE $container restarted and running ✅"
RESTARTED+=("$container")
else
error "$container restarted but crashed immediately"
notify "$container crashed after update-restart on $(hostname)" "Docker Update Remaining" "warning"
RESTART_FAILED+=("$container")
fi
else
error "Failed to restart $container after $RETRY_COUNT attempts"
notify "$container failed to restart after update on $(hostname)" "Docker Update Remaining" "warning"
RESTART_FAILED+=("$container")
fi
echo ""
done
else
log "No containers received a new image — nothing to restart"
fi
# ==============================================================================================
# ━━━ Prune Old Images ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_SYNC Pruning Dangling Images — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would prune dangling images"
PRUNED_SUMMARY="(dry run)"
else
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
echo "$PRUNED_OUTPUT" | sed 's/^/ /'
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
fi
END=$(date +%s)
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
echo ""
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINING) SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo "$ICON_CONTAINERS Scope: ${#ALL_RUNNING[@]} running — ${#EXCLUDED[@]} managed = ${#REMAINING[@]} checked"
[[ ${#UPDATED[@]} -gt 0 ]] && echo "$ICON_DONE Updated: ${UPDATED[*]}"
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && echo "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${FAILED[*]}"
[[ ${#UPDATED[@]} -gt 0 ]] && echo "$ICON_DONE New image: ${UPDATED[*]}"
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && echo "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Pull failed: ${FAILED[*]}"
[[ ${#RESTARTED[@]} -gt 0 ]] && echo "$ICON_DONE Restarted: ${RESTARTED[*]}"
[[ ${#SKIPPED_STOPPED[@]} -gt 0 ]] && echo "$ICON_WARN Not running: ${SKIPPED_STOPPED[*]} (skipped restart)"
[[ ${#RESTART_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Restart fail:${RESTART_FAILED[*]}"
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
ALL_FAILED=$(( ${#FAILED[@]} + ${#RESTART_FAILED[@]} ))
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no images pulled"
elif [[ ${#FAILED[@]} -eq 0 ]]; then
log "$ICON_DONE Status: done ✅ — ${#UPDATED[@]} updated, ${#UP_TO_DATE[@]} current"
warn "DRY RUN — no changes made"
elif [[ "$ALL_FAILED" -eq 0 ]]; then
log "$ICON_DONE Status: done ✅ — ${#RESTARTED[@]} restarted, ${#UP_TO_DATE[@]} current"
else
warn "Status: ${#FAILED[@]} pull(s) failed — containers continue on existing images"
warn "Status: $ALL_FAILED error(s) — ${#FAILED[@]} pull failure(s), ${#RESTART_FAILED[@]} restart failure(s)"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
# Pull failures are non-fatal
[[ "$ALL_FAILED" -gt 0 ]] && exit 1
exit 0
+1 -1
View File
@@ -787,7 +787,7 @@ while [[ "$WATCHDOG_RUNNING" == true ]]; do
[[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
is_skipped "$container" && continue
# Skip containers already monitored by required containers (Tier 1)
local already_required=false
already_required=false
for req in "${WATCHDOG_REQUIRED_CONTAINERS[@]:-}"; do
[[ "$container" == "$req" ]] && already_required=true && break
done
+2 -2
View File
@@ -286,7 +286,7 @@ if [[ -n "$SABNZBD_URL" ]] && [[ -n "$SABNZBD_API_KEY" ]]; then
while IFS= read -r NZO_ID; do
[[ -z "$NZO_ID" ]] && continue
JOB_TIME=$(echo "$HISTORY" | grep -A5 "$NZO_ID" | \
grep -o '"completed":[0-9]*' | grep -o '[0-9]*')
grep -o '"completed":[0-9]*' | grep -o '[0-9]*' | head -1)
[[ -z "$JOB_TIME" ]] && continue
[[ "$JOB_TIME" -gt "$CUTOFF" ]] && ((SKIPPED++)) && continue
if [[ "$DRY_RUN" == true ]]; then
@@ -332,7 +332,7 @@ if [[ -n "$SABNZBD_URL" ]] && [[ -n "$SABNZBD_API_KEY" ]]; then
while IFS= read -r NZO_ID; do
[[ -z "$NZO_ID" ]] && continue
JOB_TIME=$(echo "$FAILED_HIST" | grep -A5 "$NZO_ID" | \
grep -o '"completed":[0-9]*' | grep -o '[0-9]*')
grep -o '"completed":[0-9]*' | grep -o '[0-9]*' | head -1)
[[ -z "$JOB_TIME" ]] && continue
[[ "$JOB_TIME" -gt "$CUTOFF" ]] && ((SKIPPED++)) && continue
if [[ "$DRY_RUN" == true ]]; then