FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v2 post-migration bug fixes, lidarr_missing_art port, and Partnership SSH/FolderView3 enhancements

Browse Source 
Bug fixes across the ecosystem after v1→v2 architecture migration and Unraid 7.2.5 upgrade:

- common.sh: fix _alias_array() phantom empty element (removed [@]:-} pattern), fix
  resolve_remote_ip() with Tailscale FQDN lowercase + awk fallback, add FANART/LASTFM
  key aliases in detect_hosts(), global [@]:-} sweep across 10+ scripts
- webgui_restart.sh: fix emhttp detection (pgrep emhttpd) and restart command for 7.2.5
  (/usr/local/sbin/emhttp stop && start — rc.emhttp removed in 7.2.5)
- bandwidth_monitor.sh, continuous_scripts_status.sh: fix 'local' keyword outside function
- backup_verify.sh: fix resolve_remote_ip() called before detect_hosts()
- Orchestrators: fix script display duplication bug in status output (${entry##*/})
- rsync_stop.sh, git_pull_execute.sh, partnership_manager.sh, coffee_report: lowercase all
  tailscale ip -4 call sites to match Tailscale's lowercase device names
- master_host1.conf: fix SSH key path (gmer4lfe_rsync_automation), add FANART/LASTFM keys
- master_host2.conf: add FANART/LASTFM API keys

New: Media/lidarr_missing_art.sh
- Full ecosystem port of standalone Lidarr artwork fetcher
- Fetches missing album art via fanart.tv + Last.fm APIs
- @tsv batch extraction: 1 jq call per API response vs N*albums (8050 albums in 24s)
- HOST guard (HOST1 only), --status, --dry-run, acquire_lock

New: Initial_run/ssh_setup.sh
- Generates {hostname}_rsync_automation ed25519 keypair (skip if exists, --force to regen)
- ssh-copy-id to remote via Tailscale IP, auto-updates master_host*.conf
- --validate mode: strike tracking (SSH_MAX_STRIKES, SSH_STRIKE_RESET_HRS),
  notify at limit — Tailscale-unreachable remote does NOT count as SSH strike

New: Initial_run/partnership_onboard.sh
- Orchestrator: ssh_setup.sh then partnership_manager.sh --onboard in one command

Partnership/partnership_manager.sh: FolderView3 integration
- Derive partner folder name at runtime (strip unraid- prefix case-insensitively)
- --onboard: create {Mirror}-Failover folder with failover tier containers
- --offboard (both paths): stop + rm containers in folder, remove JSON entry
- --check: calls ssh_setup.sh --validate when IP resolves but SSH state empty
- --status: shows FolderView3 folder state and containers inline

master.conf: SSH_MAX_STRIKES, SSH_STRIKE_RESET_HRS, PARTNERSHIP_FOLDERVIEW3,
PARTNERSHIP_FOLDERVIEW3_URL added to Partnership section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Gmer4Lfe
2026-05-08 18:20:23 -04:00
co-authored by Claude Sonnet 4.6
parent 7cecda6b90
commit 5526bc1eca
24 changed files with 1108 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Docker_Essentials/docker_daily_restart.sh
+4 -4
View File
@@ -177,14 +177,14 @@ build_restart_order() {
[[ -z "$container" ]] && continue
local is_dependency=false
# Check if this container is a dependency of any other in our list
for dep_string in "${WATCHDOG_DEPENDENCIES[@]:-}"; do
for dep_string in "${WATCHDOG_DEPENDENCIES[@]}"; do
if [[ "$dep_string" == *"$container"* ]]; then
is_dependency=true
break
fi
done
# Also check associative array format
for dependent in "${!WATCHDOG_DEPENDENCIES[@]:-}"; do
for dependent in "${!WATCHDOG_DEPENDENCIES[@]}"; do
if [[ "${WATCHDOG_DEPENDENCIES[$dependent]}" == *"$container"* ]]; then
is_dependency=true
break
@@ -193,7 +193,7 @@ build_restart_order() {
if [[ "$is_dependency" == true ]]; then
# Check not already placed
local already=false
for p in "${placed[@]:-}"; do [[ "$p" == "$container" ]] && already=true && break; done
for p in "${placed[@]}"; do [[ "$p" == "$container" ]] && already=true && break; done
if [[ "$already" == false ]]; then
ORDERED_RESTART+=("$container")
placed+=("$container")
@@ -205,7 +205,7 @@ build_restart_order() {
for container in "${remaining[@]}"; do
[[ -z "$container" ]] && continue
local already=false
for p in "${placed[@]:-}"; do [[ "$p" == "$container" ]] && already=true && break; done
for p in "${placed[@]}"; do [[ "$p" == "$container" ]] && already=true && break; done
if [[ "$already" == false ]]; then
ORDERED_RESTART+=("$container")
placed+=("$container")
Docker_Essentials/docker_watchdog.sh
+1 -1
View File
@@ -131,7 +131,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_CONTAINERS Watched: ${!WATCHDOG_CONTAINERS[*]:-none}"
_watched="${!WATCHDOG_CONTAINERS[*]}"; echo "$ICON_CONTAINERS Watched: ${_watched:-none}"
echo "$ICON_CONTAINERS Required: ${WATCHDOG_REQUIRED_CONTAINERS[*]:-none}"
echo "$ICON_WATCHDOG Scan all: $WATCHDOG_SCAN_ALL"
echo "$ICON_WATCHDOG Ignore: ${WATCHDOG_SCAN_IGNORE[*]:-none}"
Docker_Essentials/docker_weekly_restart.sh
+3 -3
View File
@@ -169,7 +169,7 @@ build_restart_order() {
for container in "${remaining[@]}"; do
[[ -z "$container" ]] && continue
local is_dependency=false
for dependent in "${!WATCHDOG_DEPENDENCIES[@]:-}"; do
for dependent in "${!WATCHDOG_DEPENDENCIES[@]}"; do
if [[ "${WATCHDOG_DEPENDENCIES[$dependent]}" == *"$container"* ]]; then
is_dependency=true
break
@@ -177,7 +177,7 @@ build_restart_order() {
done
if [[ "$is_dependency" == true ]]; then
local already=false
for p in "${placed[@]:-}"; do [[ "$p" == "$container" ]] && already=true && break; done
for p in "${placed[@]}"; do [[ "$p" == "$container" ]] && already=true && break; done
if [[ "$already" == false ]]; then
ORDERED_RESTART+=("$container")
placed+=("$container")
@@ -189,7 +189,7 @@ build_restart_order() {
for container in "${remaining[@]}"; do
[[ -z "$container" ]] && continue
local already=false
for p in "${placed[@]:-}"; do [[ "$p" == "$container" ]] && already=true && break; done
for p in "${placed[@]}"; do [[ "$p" == "$container" ]] && already=true && break; done
if [[ "$already" == false ]]; then
ORDERED_RESTART+=("$container")
placed+=("$container")
Initial_run/partnership_onboard.sh
Executable
+135
View File
@@ -0,0 +1,135 @@
#!/bin/bash
# ==============================================================================================
# ============================= Partnership Onboard Setup ======================================
# ==============================================================================================
# Orchestrates the full partnership setup sequence:
# 1. SSH key setup — generate keypair, install on remote, update conf
# 2. Partnership onboard — configure auth WebUIs, write state, FolderView3 folder
#
# Run this once to join a new partner server. Both servers run their own copy.
# After setup: critical_sync_maintenance.sh carries the relationship via --check.
#
# ── PRE-REQUISITES ────────────────────────────────────────────────────────────────────────────
# Both servers must be on the same Tailscale tailnet
# Remote server must have password auth enabled for root (SSH key install step)
# PARTNERSHIP_OWNER_HOST set correctly in master.conf (HOST1 = owner by default)
#
# ── WHAT THIS SCRIPT DOES ─────────────────────────────────────────────────────────────────────
# Step 1: ssh_setup.sh — generates {hostname}_rsync_automation keypair if missing,
# installs on remote via ssh-copy-id, updates master_host*.conf
# Step 2: partnership_manager.sh --onboard
# — verifies both servers, reconfigures auth WebUIs → owner IP,
# writes ACTIVE state, creates FolderView3 folder (if enabled)
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# Initial_run/partnership_onboard.sh — full setup
# Initial_run/partnership_onboard.sh --dry-run — preview without changes
# Initial_run/partnership_onboard.sh --log — verbose output
# Initial_run/partnership_onboard.sh --skip-ssh — skip ssh_setup.sh (key already set up)
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPTS_ROOT="$SCRIPT_DIR/.."
source "$SCRIPTS_ROOT/load_config.sh"
# ── Parse --skip-ssh before parse_args ────────────────────────────────────────────────────────
SKIP_SSH=false
FILTERED_ARGS=()
for arg in "$@"; do
case "$arg" in
--skip-ssh) SKIP_SSH=true ;;
*) FILTERED_ARGS+=("$arg") ;;
esac
done
parse_args "${FILTERED_ARGS[@]}"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
detect_hosts
START=$(date +%s)
echo ""
echo "━━━ $ICON_FAILOVER Partnership Onboard Setup — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo ""
echo " This server: $MY_ID ($LOCAL_SERVER_NAME)"
echo " Partner: $REMOTE_ID ($REMOTE_SERVER_NAME)"
echo ""
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permanent changes will be made"
EXTRA_FLAGS=()
[[ "$DRY_RUN" == true ]] && EXTRA_FLAGS+=("--dry-run")
[[ "$LOG_MODE" == true ]] && EXTRA_FLAGS+=("--log")
# ==============================================================================================
# ━━━ Step 1: SSH Key Setup ━━━
# ==============================================================================================
echo ""
echo "━━━ Step 1/2 — SSH Key Setup ━━━"
if [[ "$SKIP_SSH" == true ]]; then
warn "Skipping SSH setup (--skip-ssh)"
else
if bash "$SCRIPT_DIR/ssh_setup.sh" "${EXTRA_FLAGS[@]}"; then
log "SSH key setup complete ✅"
else
error "SSH key setup failed — aborting"
error "Fix SSH key issue then re-run, or use --skip-ssh if key is already set up"
exit 1
fi
fi
# ==============================================================================================
# ━━━ Step 2: Partnership Onboard ━━━
# ==============================================================================================
echo ""
echo "━━━ Step 2/2 — Partnership Onboard ━━━"
if bash "$SCRIPTS_ROOT/Partnership/partnership_manager.sh" --onboard "${EXTRA_FLAGS[@]}"; then
log "Partnership onboard complete ✅"
ONBOARD_OK=true
else
error "Partnership onboard failed"
ONBOARD_OK=false
fi
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
END=$(date +%s)
echo ""
echo "━━━━━ $ICON_SUMMARY PARTNERSHIP SETUP SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_NET Partner: $REMOTE_SERVER_NAME"
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
echo ""
if [[ "$ONBOARD_OK" == true ]]; then
echo " Step 1 — SSH key: ✅"
echo " Step 2 — Onboard: ✅"
echo ""
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — no changes made"
else
warn "$ICON_DONE DONE — partnership established ✅"
warn "Next: verify with 'Partnership/partnership_manager.sh --status'"
fi
else
echo " Step 1 — SSH key: $( [[ "$SKIP_SSH" == true ]] && echo "skipped" || echo "✅" )"
echo " Step 2 — Onboard: ❌"
echo ""
error "Setup incomplete — resolve onboard errors above and re-run"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"
[[ "$ONBOARD_OK" == false ]] && exit 1
exit 0
Initial_run/ssh_setup.sh
Executable
+379
View File
@@ -0,0 +1,379 @@
#!/bin/bash
# ==============================================================================================
# ============================= SSH Setup ======================================================
# ==============================================================================================
# Generates the SSH keypair for rsync automation and installs it on the remote server.
# Key named after this server: hostname lowercased, unraid- prefix stripped.
# unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
# unRAID-Jayred365 → jayred365_rsync_automation
# Idempotent — skips generation if key already exists (use --force to regenerate).
# Updates master_host*.conf with key path on success.
#
# ── MODES ─────────────────────────────────────────────────────────────────────────────────────
# (default) — generate key if missing, copy to remote, update conf
# --force — regenerate key even if it exists, re-copy to remote
# --validate — test SSH auth to remote, track strikes, notify at limit
# --status — show key state, fingerprint, remote connectivity
# --dry-run — preview without creating, copying, or updating conf
#
# ── STRIKE SYSTEM (--validate) ────────────────────────────────────────────────────────────────
# Called during partnership --check cycles to detect broken SSH auth.
# Tracks consecutive SSH auth failures — not network unreachability.
# Remote Tailscale IP unreachable = network issue → not counted as SSH strike.
# Remote reachable but SSH auth fails = key issue → strike incremented.
# Strikes reset automatically after SSH_STRIKE_RESET_HRS of clean connectivity.
# At SSH_MAX_STRIKES: notify + return exit 2 (caller can escalate).
# State: DATA_DIR/ssh_strikes_{REMOTE_SERVER_NAME}.db
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# SSH_MAX_STRIKES — consecutive failures before notifying (default 5)
# SSH_STRIKE_RESET_HRS — hours since last failure before counter resets (default 24)
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# Initial_run/ssh_setup.sh — initial setup (idempotent)
# Initial_run/ssh_setup.sh --force — regenerate + re-copy
# Initial_run/ssh_setup.sh --validate — health check + strike tracking
# Initial_run/ssh_setup.sh --status — show key and connectivity state
# Any mode supports --dry-run and --log
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPTS_ROOT="$SCRIPT_DIR/.."
source "$SCRIPTS_ROOT/load_config.sh"
# ── Parse --force before parse_args ───────────────────────────────────────────────────────────
MODE="setup"
FORCE=false
FILTERED_ARGS=()
for arg in "$@"; do
case "$arg" in
--force) FORCE=true ;;
--validate) MODE="validate" ;;
--status) MODE="status" ;;
*) FILTERED_ARGS+=("$arg") ;;
esac
done
parse_args "${FILTERED_ARGS[@]}"
# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
validate_unraid_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
detect_hosts
# ── Derive key name from hostname ─────────────────────────────────────────────────────────────
# Strip unraid- prefix (case-insensitive) if present → gmer4lfe_rsync_automation
SHORT_NAME="${LOCAL_SERVER_NAME,,}"
[[ "${SHORT_NAME}" == unraid-* ]] && SHORT_NAME="${SHORT_NAME:7}"
SSH_KEY_NAME="${SHORT_NAME}_rsync_automation"
SSH_KEY_PATH="/root/.ssh/${SSH_KEY_NAME}"
SSH_PUB_PATH="${SSH_KEY_PATH}.pub"
# ── Host conf path ────────────────────────────────────────────────────────────────────────────
HOST_NUM="${MY_ID#HOST}" # "1" or "2"
HOST_CONF="$SCRIPTS_ROOT/master_host${HOST_NUM}.conf"
KEY_CONF_VAR="${MY_ID}_SSH_KEY"
# ── Strike state file ─────────────────────────────────────────────────────────────────────────
SSH_STRIKE_FILE="$DATA_DIR/ssh_strikes_${REMOTE_SERVER_NAME}.db"
# ==============================================================================================
# ─── HELPERS ──────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
test_ssh_auth() {
local remote_ip="$1"
timeout "${SSH_TIMEOUT:-15}" ssh \
-i "$SSH_KEY_PATH" \
-o BatchMode=yes \
-o ConnectTimeout="${SSH_TIMEOUT:-15}" \
-o StrictHostKeyChecking=no \
root@"$remote_ip" "echo ok" >/dev/null 2>&1
}
read_strikes() {
grep "^strikes=" "$SSH_STRIKE_FILE" 2>/dev/null | cut -d= -f2 || echo 0
}
read_last_strike_epoch() {
local ts
ts=$(grep "^last_strike=" "$SSH_STRIKE_FILE" 2>/dev/null | cut -d= -f2)
[[ -n "$ts" ]] && date -d "$ts" +%s 2>/dev/null || echo 0
}
write_strike_file() {
local strikes="$1" last_strike="${2:-}" last_success="${3:-}"
mkdir -p "$(dirname "$SSH_STRIKE_FILE")"
cat > "$SSH_STRIKE_FILE" <<EOF
strikes=${strikes}
last_strike=${last_strike}
last_success=${last_success}
updated=$(date '+%Y-%m-%d %H:%M:%S')
EOF
}
update_conf_key_path() {
if [[ ! -f "$HOST_CONF" ]]; then
warn "$(basename "$HOST_CONF") not found — update ${KEY_CONF_VAR} manually to: $SSH_KEY_PATH"
return 1
fi
local current
current=$(grep "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null | \
sed 's/.*="\?\([^"]*\)"\?.*/\1/')
if [[ "$current" == "$SSH_KEY_PATH" ]]; then
log "${KEY_CONF_VAR} already correct in $(basename "$HOST_CONF")"
return 0
fi
if grep -q "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null; then
sed -i "s|^[[:space:]]*${KEY_CONF_VAR}=.*| ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\"|" "$HOST_CONF" && \
log "${KEY_CONF_VAR} updated in $(basename "$HOST_CONF") ✅" || \
warn "Failed to update ${KEY_CONF_VAR} in $(basename "$HOST_CONF") — update manually"
else
warn "${KEY_CONF_VAR} not found in $(basename "$HOST_CONF") — add manually:"
warn " ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\""
fi
}
# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$MODE" == "status" ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY SSH SETUP STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_NET Remote: $REMOTE_SERVER_NAME"
echo ""
echo " Key name: $SSH_KEY_NAME"
echo " Key path: $SSH_KEY_PATH"
echo ""
if [[ -f "$SSH_KEY_PATH" ]]; then
local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
echo " $ICON_DONE Private key: present"
echo " Fingerprint: $local_fp"
else
echo " $ICON_ERROR Private key: NOT found"
fi
if [[ -f "$SSH_PUB_PATH" ]]; then
echo " $ICON_DONE Public key: present"
else
echo " $ICON_ERROR Public key: NOT found"
fi
# Check conf
if [[ -f "$HOST_CONF" ]]; then
current_conf=$(grep "^[[:space:]]*${KEY_CONF_VAR}=" "$HOST_CONF" 2>/dev/null | \
sed 's/.*="\?\([^"]*\)"\?.*/\1/')
if [[ "$current_conf" == "$SSH_KEY_PATH" ]]; then
echo " $ICON_DONE Conf ($(basename "$HOST_CONF")): ${KEY_CONF_VAR} ✅"
else
echo " $ICON_ERROR Conf ($(basename "$HOST_CONF")): ${KEY_CONF_VAR}=${current_conf:-not set}"
fi
fi
# Remote connectivity
echo ""
REMOTE_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -z "$REMOTE_IP" ]]; then
echo " $ICON_ERROR Remote ($REMOTE_SERVER_NAME): Tailscale unreachable"
elif [[ -f "$SSH_KEY_PATH" ]] && test_ssh_auth "$REMOTE_IP"; then
echo " $ICON_DONE Remote ($REMOTE_SERVER_NAME [$REMOTE_IP]): SSH auth OK ✅"
else
echo " $ICON_ERROR Remote ($REMOTE_SERVER_NAME [$REMOTE_IP]): SSH auth FAILED"
fi
# Strike state
if [[ -f "$SSH_STRIKE_FILE" ]]; then
echo ""
echo " Strike state:"
while IFS= read -r line; do
echo " $line"
done < "$SSH_STRIKE_FILE"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# ==============================================================================================
# ━━━ Validate (SSH health check + strike tracking) ━━━
# ==============================================================================================
if [[ "$MODE" == "validate" ]]; then
MAX_STRIKES="${SSH_MAX_STRIKES:-5}"
RESET_HRS="${SSH_STRIKE_RESET_HRS:-24}"
REMOTE_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -z "$REMOTE_IP" ]]; then
log "SSH validate — $REMOTE_SERVER_NAME Tailscale unreachable, not an SSH issue"
exit 0
fi
if [[ ! -f "$SSH_KEY_PATH" ]]; then
warn "SSH validate — key not found at $SSH_KEY_PATH — run ssh_setup.sh to create"
exit 1
fi
NOW=$(date '+%Y-%m-%d %H:%M:%S')
NOW_EPOCH=$(date +%s)
if test_ssh_auth "$REMOTE_IP"; then
# SSH works — reset strikes if they were > 0
STRIKES=$(read_strikes)
if [[ "$STRIKES" -gt 0 ]]; then
write_strike_file 0 "" "$NOW"
log "SSH validate — auth restored to $REMOTE_SERVER_NAME ✅ (strikes reset)"
else
log "SSH validate — $REMOTE_SERVER_NAME SSH auth OK ✅"
fi
exit 0
fi
# SSH auth failed — check if strikes should reset first
STRIKES=$(read_strikes)
LAST_STRIKE_EPOCH=$(read_last_strike_epoch)
RESET_SECS=$(( RESET_HRS * 3600 ))
if [[ "$STRIKES" -gt 0 ]] && \
[[ "$LAST_STRIKE_EPOCH" -gt 0 ]] && \
[[ $(( NOW_EPOCH - LAST_STRIKE_EPOCH )) -gt "$RESET_SECS" ]]; then
warn "SSH validate — strike counter reset after ${RESET_HRS}hr gap"
STRIKES=0
fi
STRIKES=$(( STRIKES + 1 ))
write_strike_file "$STRIKES" "$NOW" ""
warn "SSH validate — $REMOTE_SERVER_NAME auth FAILED (strike $STRIKES/$MAX_STRIKES)"
if [[ "$STRIKES" -ge "$MAX_STRIKES" ]]; then
error "SSH auth to $REMOTE_SERVER_NAME broken — ${STRIKES} consecutive failures"
error "Repair: run 'Initial_run/ssh_setup.sh --force' to regenerate and re-copy key"
notify "SSH auth broken to $REMOTE_SERVER_NAME ($MY_ID) — ${STRIKES} strikes, manual repair needed. Run: Initial_run/ssh_setup.sh --force" \
"SSH Setup" "warning"
exit 2
fi
exit 1
fi
# ==============================================================================================
# ━━━ Setup (default) or --force ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_GEAR SSH Key Setup — $MY_ID ($LOCAL_SERVER_NAME) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
echo ""
echo " Key name: $SSH_KEY_NAME"
echo " Key path: $SSH_KEY_PATH"
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no keys will be created or copied"
# ── Key generation ────────────────────────────────────────────────────────────────────────────
echo ""
echo "━━━ $ICON_GEAR Key Generation ━━━"
if [[ -f "$SSH_KEY_PATH" ]] && [[ "$FORCE" == false ]]; then
local_fp=$(ssh-keygen -lf "$SSH_KEY_PATH" 2>/dev/null || echo "unreadable")
log "Key already exists — skipping generation (--force to regenerate)"
log " $local_fp"
else
if [[ "$FORCE" == true ]] && [[ -f "$SSH_KEY_PATH" ]]; then
warn "Regenerating key (--force) — existing key will be replaced"
if [[ "$DRY_RUN" == false ]]; then
rm -f "$SSH_KEY_PATH" "$SSH_PUB_PATH"
fi
fi
warn "Generating ed25519 keypair: $SSH_KEY_PATH"
if [[ "$DRY_RUN" == false ]]; then
ssh-keygen -t ed25519 -N "" -f "$SSH_KEY_PATH" -C "${SSH_KEY_NAME}@${LOCAL_SERVER_NAME}" && \
log "Keypair generated ✅" || {
error "Failed to generate keypair"
exit 1
}
chmod 600 "$SSH_KEY_PATH"
chmod 644 "$SSH_PUB_PATH"
else
warn "DRY RUN — would generate: $SSH_KEY_PATH"
fi
fi
# ── Update conf ───────────────────────────────────────────────────────────────────────────────
echo ""
echo "━━━ $ICON_GEAR Update master_host${HOST_NUM}.conf ━━━"
if [[ "$DRY_RUN" == false ]]; then
update_conf_key_path
else
warn "DRY RUN — would set ${KEY_CONF_VAR}=\"${SSH_KEY_PATH}\" in master_host${HOST_NUM}.conf"
fi
# ── Copy to remote ────────────────────────────────────────────────────────────────────────────
echo ""
echo "━━━ $ICON_NET Copy Public Key to Remote ($REMOTE_SERVER_NAME) ━━━"
resolve_remote_ip
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would ssh-copy-id to root@$REMOTE_SERVER"
warn "DRY RUN — would require password for root@$REMOTE_SERVER if key auth not yet set up"
else
warn "Installing public key on $REMOTE_SERVER_NAME ($REMOTE_SERVER)..."
warn "(Password prompt for root@$REMOTE_SERVER is expected on first setup)"
if ssh-copy-id -i "$SSH_PUB_PATH" -o ConnectTimeout="${SSH_TIMEOUT:-15}" \
root@"$REMOTE_SERVER" 2>/dev/null; then
log "Public key installed on $REMOTE_SERVER_NAME ✅"
else
error "ssh-copy-id failed — check that:"
error " 1. Remote server is reachable: tailscale status"
error " 2. Password auth is enabled on remote: grep PasswordAuthentication /etc/ssh/sshd_config"
error " 3. The correct password for root@$REMOTE_SERVER_NAME is used"
exit 1
fi
fi
# ── Verify ────────────────────────────────────────────────────────────────────────────────────
echo ""
echo "━━━ $ICON_VERIFY Verify SSH Auth ━━━"
if [[ "$DRY_RUN" == false ]]; then
if test_ssh_auth "$REMOTE_SERVER"; then
log "SSH auth to $REMOTE_SERVER_NAME working ✅"
# Reset any existing strikes
if [[ -f "$SSH_STRIKE_FILE" ]]; then
write_strike_file 0 "" "$(date '+%Y-%m-%d %H:%M:%S')"
fi
else
warn "SSH auth test failed — key may need a moment to propagate"
warn "Verify manually: ssh -i $SSH_KEY_PATH root@$REMOTE_SERVER"
fi
else
warn "DRY RUN — would test: ssh -i $SSH_KEY_PATH root@$REMOTE_SERVER echo ok"
fi
# ── Summary ───────────────────────────────────────────────────────────────────────────────────
echo ""
echo "━━━━━ $ICON_SUMMARY SSH SETUP SUMMARY ━━━━━"
echo " Key: $SSH_KEY_PATH"
echo " Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
echo " Conf: ${KEY_CONF_VAR} in master_host${HOST_NUM}.conf"
echo ""
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes made" || \
warn "$ICON_DONE DONE — SSH key ready for rsync automation ✅"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
Media/lidarr_missing_art.sh
+236 -230
View File
@@ -1,78 +1,126 @@
#!/bin/bash
# ==============================================================================================
# ================================= Lidarr Missing Art =========================================
# ==============================================================================================
# Fetches missing album and artist artwork for the Lidarr music library.
# Reads from Lidarr API to discover album/artist paths, then downloads only
# what is missing — never overwrites existing files.
#
# ── SAFE DESIGN ───────────────────────────────────────────────────────────────────────────────
# READS from Lidarr only — no writes back to Lidarr
# NEVER modifies audio tags or renames media files
# NEVER overwrites existing artwork
# ONLY writes missing artwork files to existing album/artist directories
#
# ── ARTWORK TARGETS ───────────────────────────────────────────────────────────────────────────
# Album folder: cover.jpg cdart.png back.jpg
# Artist folder: folder.jpg fanart.jpg logo.png banner.jpg
#
# ── SOURCES ───────────────────────────────────────────────────────────────────────────────────
# Album covers: fanart.tv → iTunes fallback
# Artist art: fanart.tv → Deezer fallback → Last.fm fallback
#
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
# Lidarr runs on HOST1 only. detect_hosts() sets LIDARR_URL — if empty (HOST2) the
# script exits cleanly with no action rather than failing.
#
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
# acquire_lock — prevents concurrent runs during large library scans
# curl + jq check — fail fast if tools missing
# API reachability — verified before processing begins
# Skip existing — never overwrites, idempotent re-runs are safe
# Min file size — rejects corrupt/placeholder downloads (LIDARR_ART_MIN_SIZE)
# Parallel jobs — capped at LIDARR_ART_MAX_PARALLEL to avoid hammering APIs
# Download retries — LIDARR_ART_RETRIES attempts per image before giving up
# Dry-run mode — logs what would be downloaded without writing anything
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
# FANART_API_KEY — fanart.tv API key
# LASTFM_API_KEY — last.fm API key
# LIDARR_ART_MIN_SIZE — minimum valid download size in bytes
# LIDARR_ART_MAX_PARALLEL — concurrent background download jobs
# LIDARR_ART_RETRIES — download retry attempts per image
# LIDARR_ART_SLEEP_BETWEEN — seconds between fanart.tv API calls
#
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
# HOST1_LIDARR_URL — Lidarr base URL
# HOST1_LIDARR_API_KEY — Lidarr API key
# All aliased by detect_hosts() — script uses unprefixed LIDARR_URL / LIDARR_API_KEY
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
# lidarr_missing_art.sh — fetch all missing artwork
# lidarr_missing_art.sh --dry-run — preview without downloading
# lidarr_missing_art.sh --log — verbose output
# lidarr_missing_art.sh --status — show config and exit
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"
# ==============================================================================================
# ================================= Music Artwork Fetcher ======================================
# ━━━ Setup ━━━
# ==============================================================================================
#
# SAFE DESIGN
# - READS from Lidarr only
# - NEVER modifies tags
# - NEVER renames media
# - NEVER writes to Lidarr
# - ONLY writes missing artwork files
#
# ARTWORK
#
# Album Folder:
# cover.jpg
# cdart.png
# back.jpg
#
# Artist Folder:
# folder.jpg
# fanart.jpg
# logo.png
# banner.jpg
#
# SOURCES
#
# Album Covers:
# fanart.tv
# iTunes fallback
#
# Artist Art:
# fanart.tv
# Deezer fallback
# Last.fm fallback
echo ""
echo "━━━ $ICON_GEAR Setup ━━━"
# ================================= CONFIG =====================================================
if [[ "$EUID" -ne 0 ]]; then
error "Must be run as root"
exit 1
fi
success "Running as root"
if ! command -v curl >/dev/null 2>&1; then
error "curl not found — required for API calls"
exit 1
fi
if ! command -v jq >/dev/null 2>&1; then
error "jq not found — required for JSON parsing"
exit 1
fi
success "curl and jq found"
LIDARR_URL="http://localhost:8686"
LIDARR_API_KEY="b2977e71ef074bc0a0529d9fcce3b2dc"
acquire_lock
FANART_API_KEY="Yd7147a43b692df0b364b94dc47efb81"
LASTFM_API_KEY="be6dc169c33ae263e690c30d18b7491d"
detect_hosts
LOG="/tmp/artfetch.log"
# HOST guard — Lidarr runs on HOST1 only
if [[ -z "$LIDARR_URL" ]]; then
log "Lidarr not configured for $MY_ID — nothing to do"
exit 0
fi
success "Identity: $MY_ID ($LOCAL_SERVER_NAME)"
MIN_SIZE=10000
MAX_PARALLEL=4
RETRIES=2
SLEEP_BETWEEN=0.2
# ================================= FLAGS ======================================================
DRY_RUN=0
if [[ "$1" == "--dry-run" ]]; then
DRY_RUN=1
echo "Running in DRY-RUN mode"
# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_GEAR Lidarr URL: $LIDARR_URL"
echo "$ICON_NET Fanart key: $([[ -n "${FANART_API_KEY:-}" ]] && echo "set" || echo "not set")"
echo "$ICON_NET LastFM key: $([[ -n "${LASTFM_API_KEY:-}" ]] && echo "set" || echo "not set")"
echo "$ICON_GEAR Min size: ${LIDARR_ART_MIN_SIZE} bytes"
echo "$ICON_GEAR Parallel: $LIDARR_ART_MAX_PARALLEL jobs"
echo "$ICON_RETRY Retries: $LIDARR_ART_RETRIES"
echo "$ICON_TIME API sleep: ${LIDARR_ART_SLEEP_BETWEEN}s"
echo "$ICON_NOTIFY Notify: unRAID=${NOTIFY_UNRAID:-false} Discord=$([[ -n "${MY_DISCORD_WEBHOOK:-}" ]] && echo enabled || echo disabled)"
echo "$ICON_GEAR Dry Run: $DRY_RUN"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
fi
# ================================= LOGGING ====================================================
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be written"
log() {
echo "$(date '+%F %T') | $1" >> "$LOG"
}
# ================================= HELPERS ====================================================
# ==============================================================================================
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
curl_json() {
curl -s \
--connect-timeout 5 \
--max-time 20 \
"$1"
curl -s --connect-timeout 5 --max-time 20 "$1"
}
job_count() {
@@ -80,285 +128,243 @@ job_count() {
}
wait_for_slot() {
while (( $(job_count) >= MAX_PARALLEL )); do
while (( $(job_count) >= LIDARR_ART_MAX_PARALLEL )); do
sleep 0.2
done
}
# ================================= DOWNLOAD ===================================================
# Downloads URL to dest only if dest doesn't exist and downloaded size >= MIN_SIZE.
# Returns 0 on success or skip (file already exists), 1 on failure.
download_if_valid() {
local url="$1"
local dest="$2"
[[ -z "$url" || "$url" == "null" ]] && return 1
# never overwrite existing
[[ -f "$dest" ]] && return 0
if [[ "$DRY_RUN" -eq 1 ]]; then
log "[DRY] $dest"
if [[ "$DRY_RUN" == true ]]; then
info "DRY RUN — would fetch: $(basename "$dest")"
return 0
fi
for ((i=0; i<=RETRIES; i++)); do
tmp="${dest}.tmp"
curl -s \
--connect-timeout 5 \
--max-time 20 \
-L \
-o "$tmp" \
"$url"
size=$(stat -c%s "$tmp" 2>/dev/null)
if [[ "$size" -gt "$MIN_SIZE" ]]; then
local tmp="${dest}.tmp"
local i
for (( i=0; i<=LIDARR_ART_RETRIES; i++ )); do
curl -s --connect-timeout 5 --max-time 20 -L -o "$tmp" "$url"
local size
size=$(stat -c%s "$tmp" 2>/dev/null || echo 0)
if (( size > LIDARR_ART_MIN_SIZE )); then
mv "$tmp" "$dest"
log "[OK] $dest"
log "Fetched: $dest"
return 0
fi
rm -f "$tmp"
sleep 1
done
log "[FAIL] $dest"
warn "Failed to fetch valid image: $(basename "$dest")"
return 1
}
# ================================= DEEZER =====================================================
deezer_artist_image() {
local artist="$1"
local query
query=$(printf "%s" "$artist" | sed 's/ /+/g')
curl_json \
"https://api.deezer.com/search/artist?q=$query" |
curl_json "https://api.deezer.com/search/artist?q=$query" |
jq -r '.data[0].picture_xl // empty'
}
# ================================= LASTFM =====================================================
lastfm_artist_image() {
local artist="$1"
local encoded
encoded=$(printf "%s" "$artist" | sed 's/ /%20/g')
curl_json \
"https://ws.audioscrobbler.com/2.0/?method=artist.getinfo&artist=$encoded&api_key=$LASTFM_API_KEY&format=json" |
curl_json "https://ws.audioscrobbler.com/2.0/?method=artist.getinfo&artist=$encoded&api_key=$LASTFM_API_KEY&format=json" |
jq -r '.artist.image[-1]["#text"] // empty'
}
# ==============================================================================================
# ====================================== ALBUMS =================================================
# ━━━ Verify Lidarr reachable ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_NET Lidarr API ━━━"
echo "===== PROCESSING ALBUMS ====="
albums=$(curl_json \
"$LIDARR_URL/api/v1/album?apikey=$LIDARR_API_KEY")
[[ -z "$albums" || "$albums" == "null" ]] && {
echo "ERROR: Lidarr album API failed"
if ! curl_json "$LIDARR_URL/api/v1/system/status?apikey=$LIDARR_API_KEY" | jq -e '.version' >/dev/null 2>&1; then
error "Lidarr API unreachable at $LIDARR_URL — aborting"
notify "lidarr_missing_art failed — Lidarr API unreachable on $(hostname)" "Lidarr Missing Art" "warning"
exit 1
}
fi
success "Lidarr API reachable"
START=$(date +%s)
ALBUMS_CHECKED=0
ALBUMS_COMPLETE=0
ALBUM_FETCHES=0
ALBUM_FAILS=0
ARTISTS_CHECKED=0
ARTISTS_COMPLETE=0
ARTIST_FETCHES=0
ARTIST_FAILS=0
# ==============================================================================================
# ━━━ Albums ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_EMBY Albums ━━━"
albums=$(curl_json "$LIDARR_URL/api/v1/album?apikey=$LIDARR_API_KEY")
if [[ -z "$albums" || "$albums" == "null" ]]; then
error "Lidarr album API returned empty — aborting"
notify "lidarr_missing_art failed — album API empty on $(hostname)" "Lidarr Missing Art" "warning"
exit 1
fi
total_albums=$(echo "$albums" | jq '. | length')
album_count=0
info "$total_albums albums to process"
echo "$albums" | jq -c '.[]?' | while read -r album; do
while IFS=$'\t' read -r local_path mbid artist_name album_name; do
(( ALBUMS_CHECKED++ ))
((album_count++))
[[ ! -d "$local_path" ]] && continue
path=$(echo "$album" | jq -r '.path')
mbid=$(echo "$album" | jq -r '.foreignAlbumId')
artist=$(echo "$album" | jq -r '.artist.artistName')
name=$(echo "$album" | jq -r '.title')
log "[$ALBUMS_CHECKED/$total_albums] $artist_name — $album_name"
[[ ! -d "$path" ]] && continue
printf '[ALBUM %s/%s] %s - %s\n' \
"$album_count" "$total_albums" "$artist" "$name"
# skip complete albums
if [[ -f "$path/cover.jpg" &&
-f "$path/cdart.png" &&
-f "$path/back.jpg" ]]; then
echo " -> complete"
if [[ -f "$local_path/cover.jpg" &&
-f "$local_path/cdart.png" &&
-f "$local_path/back.jpg" ]]; then
(( ALBUMS_COMPLETE++ ))
log " complete — skipping"
continue
fi
wait_for_slot
(
JSON=""
if [[ -n "$mbid" && "$mbid" != "null" ]]; then
JSON=$(curl_json \
"http://webservice.fanart.tv/v3/music/albums/$mbid?api_key=$FANART_API_KEY")
sleep "$SLEEP_BETWEEN"
JSON=$(curl_json "http://webservice.fanart.tv/v3/music/albums/$mbid?api_key=$FANART_API_KEY")
sleep "$LIDARR_ART_SLEEP_BETWEEN"
fi
# ================= COVER =================
if [[ ! -f "$path/cover.jpg" ]]; then
if [[ ! -f "$local_path/cover.jpg" ]]; then
IMG=$(echo "$JSON" | jq -r '.[].albumcover[0].url // empty')
if ! download_if_valid "$IMG" "$path/cover.jpg"; then
query=$(printf "%s %s" "$artist" "$name" | sed 's/ /+/g')
itunes=$(curl_json \
"https://itunes.apple.com/search?term=$query&entity=album&limit=1" |
jq -r '.results[0].artworkUrl100 // empty' |
sed 's/100x100/600x600/')
download_if_valid "$itunes" "$path/cover.jpg"
if ! download_if_valid "$IMG" "$local_path/cover.jpg"; then
query=$(printf "%s %s" "$artist_name" "$album_name" | sed 's/ /+/g')
itunes=$(curl_json "https://itunes.apple.com/search?term=$query&entity=album&limit=1" |
jq -r '.results[0].artworkUrl100 // empty' | sed 's/100x100/600x600/')
download_if_valid "$itunes" "$local_path/cover.jpg"
fi
fi
# ================= CDART =================
if [[ ! -f "$path/cdart.png" ]]; then
if [[ ! -f "$local_path/cdart.png" ]]; then
IMG=$(echo "$JSON" | jq -r '.[].cdart[0].url // empty')
download_if_valid "$IMG" "$path/cdart.png"
download_if_valid "$IMG" "$local_path/cdart.png"
fi
# ================= BACK =================
if [[ ! -f "$path/back.jpg" ]]; then
if [[ ! -f "$local_path/back.jpg" ]]; then
IMG=$(echo "$JSON" | jq -r '.[].albumback[0].url // empty')
download_if_valid "$IMG" "$path/back.jpg"
download_if_valid "$IMG" "$local_path/back.jpg"
fi
) &
done
done < <(echo "$albums" | jq -r '.[] | [.path, .foreignAlbumId, .artist.artistName, .title] | @tsv')
wait
# ==============================================================================================
# ====================================== ARTISTS ================================================
# ━━━ Artists ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_EMBY Artists ━━━"
echo
echo "===== PROCESSING ARTISTS ====="
artists=$(curl_json "$LIDARR_URL/api/v1/artist?apikey=$LIDARR_API_KEY")
artists=$(curl_json \
"$LIDARR_URL/api/v1/artist?apikey=$LIDARR_API_KEY")
[[ -z "$artists" || "$artists" == "null" ]] && {
echo "ERROR: Lidarr artist API failed"
if [[ -z "$artists" || "$artists" == "null" ]]; then
error "Lidarr artist API returned empty — aborting"
notify "lidarr_missing_art failed — artist API empty on $(hostname)" "Lidarr Missing Art" "warning"
exit 1
}
fi
total_artists=$(echo "$artists" | jq '. | length')
artist_count=0
info "$total_artists artists to process"
echo "$artists" | jq -c '.[]?' | while read -r artist; do
while IFS=$'\t' read -r local_path mbid name; do
(( ARTISTS_CHECKED++ ))
((artist_count++))
path=$(echo "$artist" | jq -r '.path')
mbid=$(echo "$artist" | jq -r '.foreignArtistId')
name=$(echo "$artist" | jq -r '.artistName')
[[ ! -d "$path" ]] && continue
[[ ! -d "$local_path" ]] && continue
[[ -z "$mbid" || "$mbid" == "null" ]] && continue
printf '[ARTIST %s/%s] %s\n' \
"$artist_count" "$total_artists" "$name"
log "[$ARTISTS_CHECKED/$total_artists] $name"
# skip complete artists
if [[ -f "$path/folder.jpg" &&
-f "$path/fanart.jpg" &&
-f "$path/logo.png" &&
-f "$path/banner.jpg" ]]; then
echo " -> complete"
if [[ -f "$local_path/folder.jpg" &&
-f "$local_path/fanart.jpg" &&
-f "$local_path/logo.png" &&
-f "$local_path/banner.jpg" ]]; then
(( ARTISTS_COMPLETE++ ))
log " complete — skipping"
continue
fi
wait_for_slot
(
JSON=$(curl_json "http://webservice.fanart.tv/v3/music/$mbid?api_key=$FANART_API_KEY")
sleep "$LIDARR_ART_SLEEP_BETWEEN"
JSON=$(curl_json \
"http://webservice.fanart.tv/v3/music/$mbid?api_key=$FANART_API_KEY")
sleep "$SLEEP_BETWEEN"
# ================= folder.jpg =================
if [[ ! -f "$path/folder.jpg" ]]; then
if [[ ! -f "$local_path/folder.jpg" ]]; then
IMG=$(echo "$JSON" | jq -r '.artistthumb[0].url // empty')
if ! download_if_valid "$IMG" "$path/folder.jpg"; then
if ! download_if_valid "$IMG" "$local_path/folder.jpg"; then
IMG=$(deezer_artist_image "$name")
if ! download_if_valid "$IMG" "$path/folder.jpg"; then
if ! download_if_valid "$IMG" "$local_path/folder.jpg"; then
IMG=$(lastfm_artist_image "$name")
download_if_valid "$IMG" "$path/folder.jpg"
download_if_valid "$IMG" "$local_path/folder.jpg"
fi
fi
fi
# ================= fanart.jpg =================
if [[ ! -f "$path/fanart.jpg" ]]; then
if [[ ! -f "$local_path/fanart.jpg" ]]; then
IMG=$(echo "$JSON" | jq -r '.artistbackground[0].url // empty')
if ! download_if_valid "$IMG" "$path/fanart.jpg"; then
if ! download_if_valid "$IMG" "$local_path/fanart.jpg"; then
IMG=$(deezer_artist_image "$name")
download_if_valid "$IMG" "$path/fanart.jpg"
download_if_valid "$IMG" "$local_path/fanart.jpg"
fi
fi
# ================= logo.png =================
if [[ ! -f "$path/logo.png" ]]; then
if [[ ! -f "$local_path/logo.png" ]]; then
IMG=$(echo "$JSON" | jq -r '.hdmusiclogo[0].url // empty')
download_if_valid "$IMG" "$path/logo.png"
download_if_valid "$IMG" "$local_path/logo.png"
fi
# ================= banner.jpg =================
if [[ ! -f "$path/banner.jpg" ]]; then
if [[ ! -f "$local_path/banner.jpg" ]]; then
IMG=$(echo "$JSON" | jq -r '.musicbanner[0].url // empty')
download_if_valid "$IMG" "$path/banner.jpg"
download_if_valid "$IMG" "$local_path/banner.jpg"
fi
) &
done
done < <(echo "$artists" | jq -r '.[] | [.path, .foreignArtistId, .artistName] | @tsv')
wait
echo
echo "===== COMPLETE ====="
END=$(date +%s)
log "===== COMPLETE ====="
# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
echo ""
echo "━━━━━ $ICON_SUMMARY LIDARR MISSING ART SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
echo "$ICON_EMBY Albums: $ALBUMS_CHECKED checked, $ALBUMS_COMPLETE already complete"
echo "$ICON_EMBY Artists: $ARTISTS_CHECKED checked, $ARTISTS_COMPLETE already complete"
if [[ "$DRY_RUN" == true ]]; then
echo "$ICON_WARN Status: DRY RUN — no files written"
else
echo "$ICON_DONE Status: $ICON_SUCCESS ALL DONE"
notify "Lidarr art fetch complete on $(hostname) — ${ALBUMS_CHECKED} albums, ${ARTISTS_CHECKED} artists processed" "Lidarr Missing Art" "normal"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
Monitors/backup_verify.sh
+2 -1
View File
@@ -74,7 +74,6 @@ acquire_lock
# detect_hosts() sets MY_ID and aliases BACKUP_VERIFY_SHARES + DAILY_SYNC_SHARES
detect_hosts
resolve_remote_ip
# Share selection — configured list or fallback to daily sync shares
if [[ ${#BACKUP_VERIFY_SHARES[@]} -gt 0 ]]; then
@@ -120,6 +119,8 @@ fi
echo ""
echo "━━━ $ICON_SHIELD Pre-flight ━━━"
resolve_remote_ip
# Connectivity — no point making 100+ SSH calls if remote is unreachable
check_connectivity
Monitors/bandwidth_monitor.sh
+1 -1
View File
@@ -112,7 +112,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo "$ICON_BANDWIDTH Log file: $BANDWIDTH_LOG"
echo "$ICON_BANDWIDTH Retention: ${BANDWIDTH_LOG_RETENTION} days"
echo "$ICON_BANDWIDTH Warn GB: ${BANDWIDTH_WARN_GB}GB"
local entry_count=0
entry_count=0
[[ -f "$BANDWIDTH_LOG" ]] && entry_count=$(wc -l < "$BANDWIDTH_LOG")
echo "$ICON_BANDWIDTH Log entries: $entry_count"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
Monitors/continuous_scripts_status.sh
+3 -3
View File
@@ -307,8 +307,8 @@ if command -v docker >/dev/null 2>&1; then
STOPPED_FILTERED=()
while IFS= read -r name; do
[[ -z "$name" ]] && continue
local SKIP=false
for ignore in "${WATCHDOG_SCAN_IGNORE[@]:-}"; do
SKIP=false
for ignore in "${WATCHDOG_SCAN_IGNORE[@]}"; do
[[ "$name" == "$ignore" ]] && SKIP=true && break
done
[[ "$SKIP" == false ]] && STOPPED_FILTERED+=("$name")
@@ -461,7 +461,7 @@ esac
# Tailscale remote visibility — uses REMOTE_SERVER_NAME from detect_hosts()
echo ""
if command -v tailscale >/dev/null 2>&1; then
REMOTE_IP=$(tailscale ip -4 "$REMOTE_SERVER_NAME" 2>/dev/null)
REMOTE_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -n "$REMOTE_IP" ]]; then
if ping -c 1 -W 2 "$REMOTE_IP" >/dev/null 2>&1; then
echo " 🌐 $REMOTE_ID ($REMOTE_SERVER_NAME): $REMOTE_IP — reachable ✅"
Monitors/smart_health.sh
+2 -2
View File
@@ -119,7 +119,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
[[ ! -e "$drive" ]] && continue
drive_name=$(basename "$drive")
ignored=false
for ignore in "${SMART_IGNORE_DRIVES[@]:-}"; do
for ignore in "${SMART_IGNORE_DRIVES[@]}"; do
[[ "$drive_name" == "$ignore" ]] && ignored=true && break
done
if [[ "$ignored" == true ]]; then
@@ -195,7 +195,7 @@ for drive in /dev/sd? /dev/nvme?; do
# Check ignore list
ignored=false
for ignore in "${SMART_IGNORE_DRIVES[@]:-}"; do
for ignore in "${SMART_IGNORE_DRIVES[@]}"; do
[[ "$drive_name" == "$ignore" ]] && ignored=true && break
done
Monitors/zfs_memory_snapshot.sh
+1 -1
View File
@@ -81,7 +81,7 @@ detect_hosts
# Build ignore pool lookup map — O(1) check per pool
declare -A IGNORE_POOL_MAP
for pool in "${ZFS_REPORT_IGNORE_POOLS[@]:-}"; do
for pool in "${ZFS_REPORT_IGNORE_POOLS[@]}"; do
[[ -n "$pool" ]] && IGNORE_POOL_MAP["$pool"]=1
done
Orchestrators/critical_sync_maintenance.sh
+3 -3
View File
@@ -87,7 +87,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
if [[ ${#CRITICAL_SYNC_SHARES[@]} -eq 0 ]]; then
warn " No CRITICAL_SYNC_SHARES configured"
else
for share in "${CRITICAL_SYNC_SHARES[@]:-}"; do
for share in "${CRITICAL_SYNC_SHARES[@]}"; do
[[ -z "$share" ]] && continue
SHARE_PATH="${share%%|*}"
SHARE_PROFILE="${share##*|}"
@@ -102,7 +102,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
echo " None configured"
else
for entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]:-}"; do
for entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$entry" || "$entry" == \#* ]] && continue
echo " $ICON_GEAR $(basename "${entry%% *}")"
done
@@ -131,7 +131,7 @@ else
RSYNC_DRY=""
[[ "$DRY_RUN" == true ]] && RSYNC_DRY="--dry-run"
for share in "${CRITICAL_SYNC_SHARES[@]:-}"; do
for share in "${CRITICAL_SYNC_SHARES[@]}"; do
[[ -z "$share" ]] && continue
# Parse optional profile flag: "/path/to/share|profile-name"
Orchestrators/daily_sync_maintenance.sh
+7 -7
View File
@@ -100,17 +100,17 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo "$ICON_SYNC Daily enabled: ${DAILY_RSYNC_ENABLED:-false}"
echo ""
echo "━━━ Daily Sync Shares ━━━"
for share in "${DAILY_SYNC_SHARES[@]:-}"; do
for share in "${DAILY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && echo " $ICON_SYNC $share"
done
for share in "${PERSONAL_SHARES[@]:-}"; do
for share in "${PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && echo " $ICON_SYNC $share (personal)"
done
echo ""
echo "━━━ Daily Maintenance Scripts ━━━"
for entry in "${DAILY_MAINTENANCE_SCRIPTS[@]:-}"; do
for entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$entry" ]] && continue
echo " $ICON_GEAR $(basename "${entry%% *}") ${entry#* }"
echo " $ICON_GEAR ${entry##*/}"
done
echo "━━━━━━━━━━━━━━━━━━━━━━━"
exit 0
@@ -122,10 +122,10 @@ fi
# No manual HOST1/HOST2 comparison needed — aliased automatically per server
# ==============================================================================================
ALL_SHARES=()
for share in "${DAILY_SYNC_SHARES[@]:-}"; do
for share in "${DAILY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
for share in "${PERSONAL_SHARES[@]:-}"; do
for share in "${PERSONAL_SHARES[@]}"; do
[[ -n "$share" ]] && ALL_SHARES+=("$share")
done
SHARE_COUNT=${#ALL_SHARES[@]}
@@ -133,7 +133,7 @@ SHARE_COUNT=${#ALL_SHARES[@]}
# ── Split maintenance scripts: git pull runs pre-sync, rest run post-sync ─────────────────────
PRE_SYNC_SCRIPTS=()
POST_SYNC_SCRIPTS=()
for script_entry in "${DAILY_MAINTENANCE_SCRIPTS[@]:-}"; do
for script_entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$script_entry" ]] && continue
script_name=$(basename "${script_entry%% *}")
if [[ "$script_name" == "git_pull_execute.sh" ]]; then
Orchestrators/sunday_morning_coffee_report.sh
+6 -6
View File
@@ -278,7 +278,7 @@ if command -v zpool >/dev/null 2>&1; then
while IFS=$'\t' read -r pool health; do
[[ -z "$pool" ]] && continue
SKIP=false
for ignore in "${ZFS_REPORT_IGNORE_POOLS[@]:-}"; do
for ignore in "${ZFS_REPORT_IGNORE_POOLS[@]}"; do
[[ "$pool" == "$ignore" ]] && SKIP=true && break
done
[[ "$SKIP" == true ]] && continue
@@ -297,7 +297,7 @@ if command -v smartctl >/dev/null 2>&1; then
[[ ! -e "$disk" ]] && continue
DISK_NAME=$(basename "$disk")
SKIP=false
for ignore in "${SMART_IGNORE_DRIVES[@]:-}"; do
for ignore in "${SMART_IGNORE_DRIVES[@]}"; do
[[ "$DISK_NAME" == "$ignore" ]] && SKIP=true && break
done
[[ "$SKIP" == true ]] && continue
@@ -563,7 +563,7 @@ if command -v docker >/dev/null 2>&1; then
while IFS= read -r name; do
[[ -z "$name" ]] && continue
SKIP=false
for ignore in "${WATCHDOG_SCAN_IGNORE[@]:-}"; do
for ignore in "${WATCHDOG_SCAN_IGNORE[@]}"; do
[[ "$name" == "$ignore" ]] && SKIP=true && break
done
[[ "$SKIP" == false ]] && STOPPED_NAMES+=("$name")
@@ -618,7 +618,7 @@ case "$FO_STATE" in
esac
if command -v tailscale >/dev/null 2>&1; then
REMOTE_TS_IP=$(tailscale ip -4 "$REMOTE_SERVER_NAME" 2>/dev/null)
REMOTE_TS_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -n "$REMOTE_TS_IP" ]]; then
ping -c 1 -W 2 "$REMOTE_TS_IP" >/dev/null 2>&1 && \
line " 🌐 $REMOTE_ID ($REMOTE_SERVER_NAME): $REMOTE_TS_IP — reachable ✅" || \
@@ -638,7 +638,7 @@ if ! command -v openssl >/dev/null 2>&1; then
elif [[ ${#CERT_MONITOR_DOMAINS[@]} -eq 0 ]]; then
line "SSL certs: no domains configured (CERT_MONITOR_DOMAINS empty)"
else
for domain in "${CERT_MONITOR_DOMAINS[@]:-}"; do
for domain in "${CERT_MONITOR_DOMAINS[@]}"; do
[[ -z "$domain" ]] && continue
EXPIRY=$(echo | timeout "${CERT_TIMEOUT:-10}" openssl s_client \
-connect "${domain}:443" -servername "$domain" 2>/dev/null | \
@@ -757,7 +757,7 @@ if command -v smartctl >/dev/null 2>&1; then
[[ ! -e "$disk" ]] && continue
DISK_NAME=$(basename "$disk")
SKIP=false
for ignore in "${SMART_IGNORE_DRIVES[@]:-}"; do
for ignore in "${SMART_IGNORE_DRIVES[@]}"; do
[[ "$DISK_NAME" == "$ignore" ]] && SKIP=true && break
done
[[ "$SKIP" == true ]] && continue
Orchestrators/weekly_sync_maintenance.sh
+12 -12
View File
@@ -143,7 +143,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
if [[ ${#WEEKLY_SYNC_SHARES[@]} -eq 0 ]]; then
warn " No WEEKLY_SYNC_SHARES configured"
else
for share in "${WEEKLY_SYNC_SHARES[@]:-}"; do
for share in "${WEEKLY_SYNC_SHARES[@]}"; do
[[ -n "$share" ]] && echo " $ICON_SYNC $(basename "$share") ($share)"
done
fi
@@ -152,13 +152,13 @@ if [[ "$SHOW_STATUS" == true ]]; then
if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
echo " None configured"
else
for entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]:-}"; do
[[ -n "$entry" ]] && echo " $ICON_GEAR $(basename "${entry%% *}") ${entry#* }"
for entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]}"; do
[[ -n "$entry" ]] && echo " $ICON_GEAR ${entry##*/}"
done
fi
echo ""
echo "━━━ Containers (from profile config) ━━━"
for c in "${MAINTENANCE_CONTAINERS[@]:-}"; do
for c in "${MAINTENANCE_CONTAINERS[@]}"; do
[[ -n "$c" ]] && echo " $ICON_CONTAINERS $c"
done
echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -203,12 +203,12 @@ echo "━━━ $ICON_GEAR Container Updates ━━━"
if [[ "$WEEKLY_SYNC_UPDATES" == true ]]; then
if [[ "$DRY_RUN" == true ]]; then
for c in "${MAINTENANCE_CONTAINERS[@]:-}"; do
for c in "${MAINTENANCE_CONTAINERS[@]}"; do
[[ -n "$c" ]] && warn "DRY RUN — would pull: $c"
done
else
log "Pulling local container updates..."
for c in "${MAINTENANCE_CONTAINERS[@]:-}"; do
for c in "${MAINTENANCE_CONTAINERS[@]}"; do
[[ -z "$c" ]] && continue
IMAGE=$(timeout "$DOCKER_TIMEOUT" docker inspect \
"$c" --format '{{.Config.Image}}' 2>/dev/null)
@@ -233,7 +233,7 @@ if [[ "$WEEKLY_SYNC_UPDATES_REMOTE" == true ]]; then
warn "DRY RUN — would pull updates on $REMOTE_SERVER_NAME"
else
log "Pulling remote container updates on $REMOTE_SERVER_NAME..."
for c in "${MAINTENANCE_CONTAINERS[@]:-}"; do
for c in "${MAINTENANCE_CONTAINERS[@]}"; do
[[ -z "$c" ]] && continue
IMAGE=$(timeout "$SSH_TIMEOUT" ssh -i "$SSH_KEY" \
-o ConnectTimeout="$SSH_TIMEOUT" \
@@ -323,7 +323,7 @@ fi
if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
echo ""
echo "━━━ $ICON_GEAR Post-sync Jobs ━━━"
for script_entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]:-}"; do
for script_entry in "${WEEKLY_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$script_entry" ]] && continue
echo ""
run_job "$script_entry"
@@ -344,15 +344,15 @@ echo "$ICON_GEAR Updates: local=${WEEKLY_SYNC_UPDATES:-false} remote=${WEEKL
echo ""
echo "$ICON_SYNC Sync jobs ($SHARE_COUNT):"
for job in "${PASS[@]:-}"; do echo " $ICON_DONE $job"; done
for job in "${FAIL[@]:-}"; do echo " $ICON_ERROR $job"; done
for job in "${PASS[@]}"; do echo " $ICON_DONE $job"; done
for job in "${FAIL[@]}"; do echo " $ICON_ERROR $job"; done
echo " Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
echo ""
echo "$ICON_GEAR Post-sync jobs:"
for job in "${JOB_PASS[@]:-}"; do echo " $ICON_DONE $job"; done
for job in "${JOB_FAIL[@]:-}"; do echo " $ICON_ERROR $job"; done
for job in "${JOB_PASS[@]}"; do echo " $ICON_DONE $job"; done
for job in "${JOB_FAIL[@]}"; do echo " $ICON_ERROR $job"; done
fi
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
Partnership/partnership_manager.sh
+255 -18
View File
@@ -260,7 +260,7 @@ reconfigure_local_webuis() {
log "Reconfiguring local auth WebUIs → ${target_ip}..."
local failures=0
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]:-}"; do
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
[[ -z "$entry" ]] && continue
local container="${entry%%|*}"
local port="${entry##*|}"
@@ -331,11 +331,188 @@ remove_tailscale_device() {
error "Failed to remove $hostname from Tailscale — remove manually"
}
# ── FolderView3 integration ───────────────────────────────────────────────────────────────────
FOLDERVIEW3_DIR="/usr/local/emhttp/plugins/folder.view3"
FOLDERVIEW3_JSON="/boot/config/plugins/folder.view3/docker.json"
# Derive short partner name: strip unraid- prefix (case-insensitive) if present
derive_partner_folder_name() {
local hostname="$1"
local short="${hostname,,}"
[[ "$short" == unraid-* ]] && short="${short:7}"
# Capitalise first char for readability: jayred365 → Jayred365-Failover
echo "${short^}-Failover"
}
folderview3_ensure_plugin() {
if [[ -d "$FOLDERVIEW3_DIR" ]]; then
log "FolderView3 plugin present ✅"
return 0
fi
if [[ -z "${PARTNERSHIP_FOLDERVIEW3_URL:-}" ]]; then
warn "FolderView3 plugin not installed and PARTNERSHIP_FOLDERVIEW3_URL is empty"
warn "Install manually from Community Applications or set PARTNERSHIP_FOLDERVIEW3_URL in master.conf"
return 1
fi
warn "FolderView3 not found — installing from CA..."
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would run: plugin install $PARTNERSHIP_FOLDERVIEW3_URL"
return 0
fi
plugin install "$PARTNERSHIP_FOLDERVIEW3_URL" 2>/dev/null && \
log "FolderView3 installed ✅" || {
warn "FolderView3 install failed — folder will not be created"
return 1
}
}
folderview3_create_partner_folder() {
local partner_name="$1"
shift
local containers=("$@")
log "FolderView3: creating folder '$partner_name' with ${#containers[@]} container(s)..."
folderview3_ensure_plugin || return 1
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would create FolderView3 folder: $partner_name"
for c in "${containers[@]}"; do
[[ -n "$c" ]] && warn " DRY RUN — container: $c"
done
return 0
fi
# Initialise JSON if missing
if [[ ! -f "$FOLDERVIEW3_JSON" ]]; then
mkdir -p "$(dirname "$FOLDERVIEW3_JSON")"
echo '{}' > "$FOLDERVIEW3_JSON"
fi
# Check folder doesn't already exist
local existing
existing=$(jq -r --arg name "$partner_name" \
'to_entries[] | select(.value.name == $name) | .key' \
"$FOLDERVIEW3_JSON" 2>/dev/null)
if [[ -n "$existing" ]]; then
log "FolderView3: folder '$partner_name' already exists (id: $existing) — skipping"
return 0
fi
# Build containers JSON array
local containers_json
containers_json=$(printf '%s\n' "${containers[@]}" | \
grep -v '^$' | jq -R . | jq -s .)
# Generate a stable random ID from timestamp+name
local folder_id
folder_id=$(echo "${partner_name}$(date +%s%N)" | md5sum | head -c 12)
jq --arg id "$folder_id" --arg name "$partner_name" \
--argjson containers "$containers_json" \
'.[$id] = {"name": $name, "containers": $containers, "containerImages": {}}' \
"$FOLDERVIEW3_JSON" > "${FOLDERVIEW3_JSON}.tmp" && \
mv "${FOLDERVIEW3_JSON}.tmp" "$FOLDERVIEW3_JSON" && \
log "FolderView3: folder '$partner_name' created with ${#containers[@]} container(s) ✅" || {
warn "FolderView3: failed to write JSON — check $FOLDERVIEW3_JSON"
return 1
}
}
folderview3_remove_partner_folder() {
local partner_name="$1"
log "FolderView3: removing folder '$partner_name' and stopping its containers..."
if [[ ! -f "$FOLDERVIEW3_JSON" ]]; then
log "FolderView3: JSON not found — nothing to remove"
return 0
fi
if ! command -v jq >/dev/null 2>&1; then
warn "jq not found — cannot manage FolderView3 JSON"
return 1
fi
# Get containers from this folder
local containers_json
containers_json=$(jq -r --arg name "$partner_name" \
'to_entries[] | select(.value.name == $name) | .value.containers[]' \
"$FOLDERVIEW3_JSON" 2>/dev/null)
if [[ -z "$containers_json" ]]; then
log "FolderView3: folder '$partner_name' not found — nothing to remove"
return 0
fi
# Stop and remove each container in the folder
local stopped=0 removed=0 failed=0
while IFS= read -r container; do
[[ -z "$container" ]] && continue
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would stop + rm: $container"
continue
fi
# Stop if running
if timeout "${DOCKER_TIMEOUT:-30}" docker inspect "$container" >/dev/null 2>&1; then
if timeout "${DOCKER_TIMEOUT:-30}" docker stop "$container" >/dev/null 2>&1; then
log "$container stopped ✅"
(( stopped++ ))
else
warn "$container stop failed"
(( failed++ ))
fi
if timeout "${DOCKER_TIMEOUT:-30}" docker rm "$container" >/dev/null 2>&1; then
log "$container removed ✅"
(( removed++ ))
else
warn "$container rm failed (may not exist)"
fi
else
log "$container not found locally — skipping"
fi
done <<< "$containers_json"
# Remove folder entry from JSON
if [[ "$DRY_RUN" == false ]]; then
jq --arg name "$partner_name" \
'with_entries(select(.value.name != $name))' \
"$FOLDERVIEW3_JSON" > "${FOLDERVIEW3_JSON}.tmp" && \
mv "${FOLDERVIEW3_JSON}.tmp" "$FOLDERVIEW3_JSON" && \
log "FolderView3: folder '$partner_name' removed ✅" || \
warn "FolderView3: failed to remove folder from JSON"
else
warn "DRY RUN — would remove folder '$partner_name' from $FOLDERVIEW3_JSON"
fi
[[ "$DRY_RUN" == false ]] && \
log "FolderView3 cleanup: $stopped stopped, $removed removed, $failed failed"
return 0
}
# Gather all partner failover containers for this server (all tiers)
gather_partner_failover_containers() {
local out_var="$1"
eval "${out_var}=()"
local tier var
for tier in TIER1 TIER2 TIER3 TIER4; do
var="FAILOVER_${MY_ID}_RUNS_FOR_${REMOTE_ID}_${tier}"
# Check if var is set and is an array
if declare -p "$var" >/dev/null 2>&1; then
local -a _tmp_arr
eval "_tmp_arr=(\"\${${var}[@]}\")"
for c in "${_tmp_arr[@]}"; do
[[ -n "$c" ]] && eval "${out_var}+=(\"\$c\")"
done
fi
done
}
check_both_healthy() {
mountpoint -q /mnt/user 2>/dev/null || { error "Local array not healthy"; return 1; }
local mirror_ip
mirror_ip=$(tailscale ip -4 "$MIRROR" 2>/dev/null)
mirror_ip=$(tailscale ip -4 "${MIRROR,,}" 2>/dev/null)
[[ -z "$mirror_ip" ]] && { error "Cannot resolve $MIRROR Tailscale IP"; return 1; }
timeout "$SSH_TIMEOUT" ssh -i "$MIRROR_SSH_KEY" \
@@ -387,8 +564,8 @@ update_master_conf() {
if [[ "$MODE" == "status" ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY PARTNERSHIP STATUS ━━━━━"
OWNER_IP=$(tailscale ip -4 "$OWNER" 2>/dev/null || echo "unreachable")
MIRROR_IP=$(tailscale ip -4 "$MIRROR" 2>/dev/null || echo "unreachable")
OWNER_IP=$(tailscale ip -4 "${OWNER,,}" 2>/dev/null || echo "unreachable")
MIRROR_IP=$(tailscale ip -4 "${MIRROR,,}" 2>/dev/null || echo "unreachable")
echo " $ICON_HOST My ID: $MY_ID ($LOCAL_SERVER_NAME)"
echo " Owner: $OWNER_ID ($OWNER — $OWNER_IP)"
echo " Mirror: $MIRROR_ID ($MIRROR — $MIRROR_IP)"
@@ -413,7 +590,7 @@ if [[ "$MODE" == "status" ]]; then
echo ""
# Remote state
REMOTE_IP=$(tailscale ip -4 "$REMOTE_SERVER_NAME" 2>/dev/null)
REMOTE_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -n "$REMOTE_IP" ]]; then
REMOTE_CONTENT=$(read_remote_state "$REMOTE_IP" "$SSH_KEY" "$REMOTE_STATE_FILE")
if [[ -n "$REMOTE_CONTENT" ]]; then
@@ -437,11 +614,32 @@ if [[ "$MODE" == "status" ]]; then
echo ""
echo " Auth WebUIs:"
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]:-}"; do
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
[[ -z "$entry" ]] && continue
echo " ${entry%%|*} → port ${entry##*|}"
done
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
echo ""
FOLDER_STATUS_NAME=$(derive_partner_folder_name "$REMOTE_SERVER_NAME")
if [[ -f "$FOLDERVIEW3_JSON" ]]; then
FOLDER_EXISTS=$(jq -r --arg name "$FOLDER_STATUS_NAME" \
'to_entries[] | select(.value.name == $name) | .key' \
"$FOLDERVIEW3_JSON" 2>/dev/null)
if [[ -n "$FOLDER_EXISTS" ]]; then
FOLDER_CONTAINERS=$(jq -r --arg name "$FOLDER_STATUS_NAME" \
'[to_entries[] | select(.value.name == $name) | .value.containers[]] | join(", ")' \
"$FOLDERVIEW3_JSON" 2>/dev/null)
echo " FolderView3: $FOLDER_STATUS_NAME ✅"
echo " Containers: $FOLDER_CONTAINERS"
else
echo " FolderView3: $FOLDER_STATUS_NAME — not found in JSON"
fi
else
echo " FolderView3: config not found ($FOLDERVIEW3_JSON)"
fi
fi
if [[ -f "$OFFLINE_COUNTER" ]]; then
OFFLINE_DAYS=$(cat "$OFFLINE_COUNTER" 2>/dev/null || echo 0)
[[ "$OFFLINE_DAYS" -gt 0 ]] && \
@@ -484,14 +682,21 @@ if [[ "$MODE" == "check" ]]; then
fi
# Read remote state file
REMOTE_IP=$(tailscale ip -4 "$REMOTE_SERVER_NAME" 2>/dev/null)
REMOTE_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -z "$REMOTE_IP" ]]; then
log "Partnership check — remote unreachable, skipping state check"
exit 0
fi
REMOTE_CONTENT=$(read_remote_state "$REMOTE_IP" "$SSH_KEY" "$REMOTE_STATE_FILE")
[[ -z "$REMOTE_CONTENT" ]] && log "Partnership check — remote state file not found" && exit 0
if [[ -z "$REMOTE_CONTENT" ]]; then
# IP resolved but SSH returned nothing — could be auth failure, not just missing file
if [[ -f "$SCRIPT_DIR/../Initial_run/ssh_setup.sh" ]]; then
bash "$SCRIPT_DIR/../Initial_run/ssh_setup.sh" --validate 2>/dev/null || true
fi
log "Partnership check — remote state file not found"
exit 0
fi
REMOTE_STATE=$(echo "$REMOTE_CONTENT" | grep "^state=" | cut -d= -f2)
LOCAL_STATE=$(read_state_file "$LOCAL_STATE_FILE" "state" 2>/dev/null || echo "UNKNOWN")
@@ -587,8 +792,8 @@ if [[ "$MODE" == "onboard" ]]; then
check_remote_array || exit 1
check_remote_docker_daemon || exit 1
OWNER_IP=$(tailscale ip -4 "$OWNER" 2>/dev/null)
MIRROR_IP=$(tailscale ip -4 "$MIRROR" 2>/dev/null)
OWNER_IP=$(tailscale ip -4 "${OWNER,,}" 2>/dev/null)
MIRROR_IP=$(tailscale ip -4 "${MIRROR,,}" 2>/dev/null)
[[ -z "$OWNER_IP" ]] && { error "Cannot resolve $OWNER Tailscale IP"; exit 1; }
[[ -z "$MIRROR_IP" ]] && { error "Cannot resolve $MIRROR Tailscale IP"; exit 1; }
@@ -601,7 +806,7 @@ if [[ "$MODE" == "onboard" ]]; then
echo "━━━ $ICON_CONTAINERS Reconfigure Mirror WebUIs → $OWNER_IP ━━━"
WEBUI_FAILURES=0
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]:-}"; do
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
[[ -z "$entry" ]] && continue
container="${entry%%|*}"
port="${entry##*|}"
@@ -613,7 +818,7 @@ if [[ "$MODE" == "onboard" ]]; then
if [[ "${PARTNERSHIP_ONBOARD_VERIFY:-true}" == true ]]; then
echo ""
echo "━━━ $ICON_VERIFY WebUI Connectivity ━━━"
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]:-}"; do
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
[[ -z "$entry" ]] && continue
container="${entry%%|*}"
port="${entry##*|}"
@@ -636,6 +841,20 @@ if [[ "$MODE" == "onboard" ]]; then
push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
echo "0" > "$OFFLINE_COUNTER"
# FolderView3 — create partner folder with this server's failover containers for remote
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
echo ""
echo "━━━ $ICON_CONTAINERS FolderView3 Integration ━━━"
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$MIRROR")
declare -a PARTNER_CONTAINERS=()
gather_partner_failover_containers PARTNER_CONTAINERS
if [[ ${#PARTNER_CONTAINERS[@]} -gt 0 ]]; then
folderview3_create_partner_folder "$PARTNER_FOLDER_NAME" "${PARTNER_CONTAINERS[@]}"
else
log "No FAILOVER_${MY_ID}_RUNS_FOR_${REMOTE_ID}_TIER* containers configured — skipping folder creation"
fi
fi
# Summary
echo ""
echo "━━━━━ $ICON_SUMMARY ONBOARD SUMMARY ━━━━━"
@@ -692,12 +911,20 @@ if [[ "$MODE" == "offboard" ]]; then
echo "━━━ $ICON_CONTAINERS Reconfigure Local WebUIs → localhost ━━━"
reconfigure_local_webuis "localhost"
# FolderView3 — remove partner folder and clean containers
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
echo ""
echo "━━━ $ICON_CONTAINERS FolderView3 Cleanup ━━━"
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$OWNER")
folderview3_remove_partner_folder "$PARTNER_FOLDER_NAME"
fi
NOW=$(date '+%Y-%m-%d %H:%M:%S')
write_state_file "$LOCAL_STATE_FILE" \
"INACTIVE" "" "$NOW" "$LOCAL_SERVER_NAME" "$REASON"
log "Local state: INACTIVE ✅"
OWNER_IP=$(tailscale ip -4 "$OWNER" 2>/dev/null)
OWNER_IP=$(tailscale ip -4 "${OWNER,,}" 2>/dev/null)
if [[ -n "$OWNER_IP" ]]; then
push_state_to_remote "$LOCAL_STATE_FILE" "$OWNER_IP" "$MIRROR_SSH_KEY"
notify "Partnership offboard requested by $MIRROR — $OWNER will finalise on next check" \
@@ -738,7 +965,7 @@ if [[ "$MODE" == "offboard" ]]; then
echo "━━━ $ICON_SYNC Final Sync ━━━"
do_final_sync
MIRROR_IP=$(tailscale ip -4 "$MIRROR" 2>/dev/null)
MIRROR_IP=$(tailscale ip -4 "${MIRROR,,}" 2>/dev/null)
MIRROR_REACHABLE=false
[[ -n "$MIRROR_IP" ]] && MIRROR_REACHABLE=true
@@ -748,7 +975,7 @@ if [[ "$MODE" == "offboard" ]]; then
WEBUI_FAILURES=0
if [[ "$MIRROR_REACHABLE" == true ]]; then
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]:-}"; do
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
[[ -z "$entry" ]] && continue
container="${entry%%|*}"
port="${entry##*|}"
@@ -784,6 +1011,14 @@ if [[ "$MODE" == "offboard" ]]; then
push_state_to_remote "$LOCAL_STATE_FILE" "$MIRROR_IP" "$MIRROR_SSH_KEY"
fi
# FolderView3 — remove partner folder and clean containers on this (owner) side
if [[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]]; then
echo ""
echo "━━━ $ICON_CONTAINERS FolderView3 Cleanup ━━━"
PARTNER_FOLDER_NAME=$(derive_partner_folder_name "$MIRROR")
folderview3_remove_partner_folder "$PARTNER_FOLDER_NAME"
fi
# Tailscale removal
if [[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]]; then
echo ""
@@ -822,6 +1057,8 @@ if [[ "$MODE" == "offboard" ]]; then
echo " WebUI failures: $WEBUI_FAILURES"
echo " Critical rsync: disabled ✅"
echo " State: INACTIVE ✅"
[[ "${PARTNERSHIP_FOLDERVIEW3:-false}" == true ]] && \
echo " FolderView3: ${PARTNER_FOLDER_NAME:-} cleaned ✅"
[[ "${PARTNERSHIP_REMOVE_TAILSCALE:-true}" == true ]] && \
echo " Tailscale: $MIRROR removed ✅"
echo ""
@@ -912,8 +1149,8 @@ if [[ "$MODE" == "transfer" ]]; then
NEW_OWNER_SSH_KEY="${!NEW_OWNER_SSH_KEY_VAR}"
NEW_MIRROR_SSH_KEY="${!NEW_MIRROR_SSH_KEY_VAR}"
NEW_OWNER_IP=$(tailscale ip -4 "$NEW_OWNER" 2>/dev/null)
NEW_MIRROR_IP=$(tailscale ip -4 "$NEW_MIRROR" 2>/dev/null)
NEW_OWNER_IP=$(tailscale ip -4 "${NEW_OWNER,,}" 2>/dev/null)
NEW_MIRROR_IP=$(tailscale ip -4 "${NEW_MIRROR,,}" 2>/dev/null)
[[ -z "$NEW_OWNER_IP" ]] && { error "Cannot resolve new owner Tailscale IP"; exit 1; }
[[ -z "$NEW_MIRROR_IP" ]] && { error "Cannot resolve new mirror Tailscale IP"; exit 1; }
@@ -929,7 +1166,7 @@ if [[ "$MODE" == "transfer" ]]; then
log "New mirror ($NEW_MIRROR) WebUIs → new owner ($NEW_OWNER_IP)"
WEBUI_FAILURES=0
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]:-}"; do
for entry in "${PARTNERSHIP_AUTH_WEBUIS[@]}"; do
[[ -z "$entry" ]] && continue
container="${entry%%|*}"
port="${entry##*|}"
common.sh
+16 -5
View File
@@ -506,6 +506,8 @@ detect_hosts() {
LIDARR_URL_VAR="${MY_ID}_LIDARR_URL"; LIDARR_URL="${!LIDARR_URL_VAR:-}"
LIDARR_API_KEY_VAR="${MY_ID}_LIDARR_API_KEY"; LIDARR_API_KEY="${!LIDARR_API_KEY_VAR:-}"
LIDARR_MUSIC_ROOT_VAR="${MY_ID}_LIDARR_MUSIC_ROOT"; LIDARR_MUSIC_ROOT="${!LIDARR_MUSIC_ROOT_VAR:-}"
FANART_API_KEY_VAR="${MY_ID}_FANART_API_KEY"; FANART_API_KEY="${!FANART_API_KEY_VAR:-}"
LASTFM_API_KEY_VAR="${MY_ID}_LASTFM_API_KEY"; LASTFM_API_KEY="${!LASTFM_API_KEY_VAR:-}"
SONARR_URL_VAR="${MY_ID}_SONARR_URL"; SONARR_URL="${!SONARR_URL_VAR:-}"
SONARR_API_KEY_VAR="${MY_ID}_SONARR_API_KEY"; SONARR_API_KEY="${!SONARR_API_KEY_VAR:-}"
@@ -533,7 +535,7 @@ detect_hosts() {
_alias_array() {
local alias_name="$1"
local source_var="${MY_ID}_${alias_name}"
eval "${alias_name}=(\"\${${source_var}[@]:-}\")"
eval "${alias_name}=(\"\${${source_var}[@]}\")"
}
_alias_array "DAILY_SYNC_SHARES"
@@ -590,10 +592,19 @@ detect_hosts() {
resolve_remote_ip() {
log "Resolving remote IP for $REMOTE_SERVER_NAME..."
REMOTE_SERVER=$(tailscale ip -4 "$REMOTE_SERVER_NAME" 2>/dev/null)
local ts_name="${REMOTE_SERVER_NAME,,}"
# Direct lookup — works when MagicDNS short-name resolution is active
REMOTE_SERVER=$(tailscale ip -4 "$ts_name" 2>/dev/null)
# Fallback — parse tailscale status for FQDN entries (e.g. hostname.tailXXXX.ts.net)
if [[ -z "$REMOTE_SERVER" ]]; then
REMOTE_SERVER=$(tailscale status 2>/dev/null | awk -v name="$ts_name" '$2 ~ "^" name { print $1; exit }')
fi
if [[ -z "$REMOTE_SERVER" ]]; then
error "Failed to resolve Tailscale IP for $REMOTE_SERVER_NAME"
error "Check: tailscale status | grep $REMOTE_SERVER_NAME"
error "Check: tailscale status | grep $ts_name"
exit 1
fi
info "$ICON_NET Remote IP: $REMOTE_SERVER"
@@ -1105,7 +1116,7 @@ start_containers() {
for c in "${RUNNING_CONTAINERS[@]}"; do
[[ -z "$c" ]] && continue
local needs_delay=false
for d in "${DELAYED_CONTAINERS[@]:-}"; do
for d in "${DELAYED_CONTAINERS[@]}"; do
[[ "$c" == "$d" ]] && needs_delay=true && break
done
if [[ "$needs_delay" == true ]]; then
@@ -1133,7 +1144,7 @@ start_local_containers() {
for c in "${LOCAL_RUNNING_CONTAINERS[@]}"; do
[[ -z "$c" ]] && continue
local needs_delay=false
for d in "${DELAYED_CONTAINERS[@]:-}"; do
for d in "${DELAYED_CONTAINERS[@]}"; do
[[ "$c" == "$d" ]] && needs_delay=true && break
done
if [[ "$needs_delay" == true ]]; then
git_pull_execute.sh
+1 -1
View File
@@ -82,7 +82,7 @@ if docker ps --format "{{.Names}}" 2>/dev/null | grep -q "^${GITEA_CONTAINER}$";
else
# Gitea not running locally — find it on the remote server via Tailscale
log "Gitea not running locally — checking remote server"
GITEA_IP=$(tailscale ip -4 "$REMOTE_SERVER_NAME" 2>/dev/null)
GITEA_IP=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -n "$GITEA_IP" ]]; then
info "$ICON_NET Gitea on $REMOTE_SERVER_NAME — connecting via Tailscale $GITEA_IP"
elif [[ -n "${GITEA_DOMAIN:-}" ]]; then
master.conf
+21
View File
@@ -178,6 +178,20 @@
PARTNERSHIP_ONBOARD_NOTIFY=true # notify both servers on completion
PARTNERSHIP_SYNC_INTERVAL=15 # minutes — informational, actual schedule in cron
# SSH key lifecycle — managed by Initial_run/ssh_setup.sh.
# Key named after this server: hostname lowercased, unraid- prefix stripped.
# unRAID-Gmer4Lfe → /root/.ssh/gmer4lfe_rsync_automation
# Run Initial_run/partnership_onboard.sh to generate, copy, and update conf automatically.
SSH_MAX_STRIKES=5 # consecutive SSH auth failures before critical notify
SSH_STRIKE_RESET_HRS=24 # hours since last failure before strike counter resets
# FolderView3 plugin integration — chodeus/VladoPortos/scolcipitato.
# Creates a "{Partner}-Failover" folder on onboard, removes + cleans containers on offboard.
# Folder name derived from remote hostname (strip unraid- prefix case-insensitively).
# Plugin config: /boot/config/plugins/folder.view3/docker.json
PARTNERSHIP_FOLDERVIEW3=true # create/remove FolderView3 folder on onboard/offboard
PARTNERSHIP_FOLDERVIEW3_URL="" # CA plugin URL — leave empty to skip auto-install
# ==============================================================================================
# ── LOGGING ───────────────────────────────────────────────────────────────────────────────────
# ==============================================================================================
@@ -835,6 +849,13 @@
"*.lrc"
)
# Lidarr artwork fetcher settings
LIDARR_ART_MIN_SIZE=10000 # bytes — reject downloads smaller than this
LIDARR_ART_MAX_PARALLEL=4 # concurrent background download jobs
LIDARR_ART_RETRIES=2 # download retry attempts per image
LIDARR_ART_SLEEP_BETWEEN=0.2 # seconds between fanart.tv API calls
# HOST*_FANART_API_KEY / HOST*_LASTFM_API_KEY — set in master_host*.conf
# Sonarr shared settings
SONARR_ORPHAN_AGE=7 # days — files must be older than this before eligible for deletion
SONARR_MAX_DELETE_GB=10 # require --i-know-what-im-doing if deletion exceeds this
master_host1.conf
+3 -1
View File
@@ -72,7 +72,7 @@
# SSH key used for all server-to-server operations — rsync, failover container commands.
# Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys.
HOST1_SSH_KEY="root/.ssh/gmer4lfe_rsync_automation"
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
# ━━━ Emby ━━━
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
@@ -495,6 +495,8 @@
HOST1_LIDARR_URL="http://localhost:8686"
HOST1_LIDARR_API_KEY="b2977e71ef074bc0a0529d9fcce3b2dc"
HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music-New"
HOST1_FANART_API_KEY="Yd7147a43b692df0b364b94dc47efb81"
HOST1_LASTFM_API_KEY="be6dc169c33ae263e690c30d18b7491d"
declare -A HOST1_LIDARR_PATH_MAP=(
["/ext-music"]="/mnt/user/Music-New"
master_host2.conf
+2
View File
@@ -417,6 +417,8 @@
# PATH MAPS — container path → host path translation.
# Arr stores file paths using container-internal paths — scripts need host paths to scan.
# Add one entry per root folder in arr Settings → Media Management → Root Folders.
HOST2_FANART_API_KEY="Yd7147a43b692df0b364b94dc47efb81"
HOST2_LASTFM_API_KEY="be6dc169c33ae263e690c30d18b7491d"
# ━━━ Sonarr ━━━
HOST2_SONARR_URL="http://localhost:8989"
unRAID_Essentials/rsync_stop.sh
+5 -3
View File
@@ -84,11 +84,13 @@ validate_unraid_cmd \
acquire_lock
detect_hosts
resolve_remote_ip
# Remote reachability
# Soft IP resolution — rsync_stop continues local-only if remote unreachable
REMOTE_REACHABLE=false
if timeout "$SSH_TIMEOUT" ping -c1 -W3 "$REMOTE_SERVER" &>/dev/null; then
REMOTE_SERVER=$(tailscale ip -4 "${REMOTE_SERVER_NAME,,}" 2>/dev/null)
if [[ -z "$REMOTE_SERVER" ]]; then
warn "$REMOTE_SERVER_NAME — cannot resolve Tailscale IP, remote operations will be skipped"
elif timeout "$SSH_TIMEOUT" ping -c1 -W3 "$REMOTE_SERVER" &>/dev/null; then
REMOTE_REACHABLE=true
log "$REMOTE_SERVER_NAME reachable ✅"
else
unRAID_Essentials/user_scripts_stop.sh
+1 -1
View File
@@ -103,7 +103,7 @@ get_user_script_pids() {
fi
done
)
printf '%s\n' "${pids[@]}"
(( ${#pids[@]} > 0 )) && printf '%s\n' "${pids[@]}"
}
# ==============================================================================================
unRAID_Essentials/webgui_restart.sh
+4 -4
View File
@@ -106,7 +106,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo " $ICON_SUCCESS php-fpm: running ($FPM_COUNT workers) ✅" || \
echo " $ICON_ERROR php-fpm: NOT running"
pgrep -x emhttp >/dev/null 2>&1 && \
pgrep emhttpd >/dev/null 2>&1 && \
echo " $ICON_SUCCESS emhttp: running ✅" || \
echo " $ICON_ERROR emhttp: NOT running"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -209,9 +209,9 @@ if [[ "$RECOVERY_OK" == false ]]; then
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart emhttp"
else
if /etc/rc.d/rc.emhttp restart >/dev/null 2>&1; then
if /usr/local/sbin/emhttp stop >/dev/null 2>&1 && /usr/local/sbin/emhttp start >/dev/null 2>&1; then
sleep 2
if pgrep -x emhttp >/dev/null 2>&1; then
if pgrep emhttpd >/dev/null 2>&1; then
warn "emhttp restarted ✅"
else
error "emhttp not running after restart command"
@@ -255,7 +255,7 @@ if [[ "$RECOVERY_OK" == true ]]; then
else
echo "$ICON_ERROR Status: UNRECOVERED — all three restart steps failed"
echo "$ICON_ERROR Manual intervention needed:"
echo " 1. Check: pgrep -x nginx emhttp"
echo " 1. Check: pgrep nginx; pgrep emhttpd"
echo " 2. Check: journalctl -u nginx --since '10 minutes ago'"
echo " 3. Try: server_reboot.sh if nothing else works"
notify "WebGUI UNRECOVERED on $(hostname) ($MY_ID) — nginx + php-fpm + emhttp restart all failed — manual intervention needed" \