FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Standardize orchestrator child-script execution and logging

Browse Source 
Every orchestrator invoked its children differently — four near-duplicate
run_job() copies, a differently-shaped run_watchdog(), or plain inline bash
calls, each with its own take on path resolution, pass/fail naming, and
dry-run threading. Extracted one shared run_orch_child() into common.sh so
there's a single place to fix or extend this behavior going forward.

Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were
checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log
silently did nothing beyond basic logging on those two. Fixed to
$ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no
trailing exit, so their exit codes reflected whatever the last command
happened to return rather than actual success/failure. transcode_management.sh
had no failure notification and no summary at all. Also made
transcode_management.sh's two-script pipeline config-driven
(TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room
to extend it later without editing the orchestrator itself.
This commit is contained in:
Gmer4Lfe
2026-07-03 10:57:52 -04:00
parent 6fd22ae4ee
commit 2a062e5140
14 changed files with 303 additions and 370 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Orchestrators/critical_sync_maintenance.sh
+16 -29
View File
@@ -80,6 +80,7 @@
# ==============================================================================================
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
@@ -199,29 +200,12 @@ fi
# ==============================================================================================
# ━━━ Critical Maintenance Scripts ━━━
# ==============================================================================================
if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -gt 0 ]]; then
for script_entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$script_entry" || "$script_entry" == \#* ]] && continue
SCRIPT_PATH="$SCRIPT_DIR/../${script_entry%% *}"
SCRIPT_ARGS="${script_entry#* }"
[[ "$SCRIPT_ARGS" == "$script_entry" ]] && SCRIPT_ARGS=""
[[ "$DRY_RUN" == true ]] && SCRIPT_ARGS="$SCRIPT_ARGS --dry-run"
SCRIPT_NAME=$(basename "$SCRIPT_PATH")
if [[ ! -f "$SCRIPT_PATH" ]]; then
warn "$SCRIPT_NAME not found at $SCRIPT_PATH — skipping"
continue
fi
log "Running: $SCRIPT_NAME"
bash "$SCRIPT_PATH" $SCRIPT_ARGS
EXIT_CODE=$?
[[ "$EXIT_CODE" -ne 0 ]] && \
warn "$SCRIPT_NAME exited with code $EXIT_CODE"
done
fi
JOB_PASS=()
JOB_FAIL=()
for script_entry in "${CRITICAL_MAINTENANCE_SCRIPTS[@]}"; do
[[ -z "$script_entry" || "$script_entry" == \#* ]] && continue
run_orch_child "$script_entry"
done
# ==============================================================================================
# ━━━ Partnership Check ━━━
@@ -247,20 +231,23 @@ fi
END=$(date +%s)
DURATION=$(format_duration $(( END - START )))
# Silent when healthy — only show summary if there were failures or notable events
if [[ ${#FAIL[@]} -gt 0 ]]; then
TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
# Minimal one-liner when healthy — 30-min cadence, keep it quiet. Full detail on failure.
if [[ "$TOTAL_FAIL" -gt 0 ]]; then
echo ""
echo "━━━━━ $ICON_SUMMARY CRITICAL SYNC SUMMARY ━━━━━"
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME Duration: $DURATION"
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
echo "$ICON_ERROR Failed: ${FAIL[*]}"
[[ ${#PASS[@]} -gt 0 ]] && echo "Synced: ${PASS[*]}"
[[ ${#FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed shares: ${FAIL[*]}"
[[ ${#JOB_FAIL[@]} -gt 0 ]] && echo "$ICON_ERROR Failed jobs: ${JOB_FAIL[*]}"
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
notify "Critical sync failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
notify "Critical sync failed on $(hostname) ($MY_ID) — ${FAIL[*]} ${JOB_FAIL[*]}" \
"Critical Sync" "warning"
exit 1
else
echo "Critical sync complete — $MY_ID — ${DURATION} — ${#PASS[@]} share(s)"
echo "Critical sync complete — $MY_ID — ${DURATION} — ${#PASS[@]} share(s), ${#JOB_PASS[@]} job(s)"
fi
exit 0