Standardize orchestrator child-script execution and logging
Every orchestrator invoked its children differently — four near-duplicate run_job() copies, a differently-shaped run_watchdog(), or plain inline bash calls, each with its own take on path resolution, pass/fail naming, and dry-run threading. Extracted one shared run_orch_child() into common.sh so there's a single place to fix or extend this behavior going forward. Along the way: watchdog_orchestrator.sh and monthly_maintenance.sh were checking $VERBOSE, a variable nothing in the codebase ever assigns, so --log silently did nothing beyond basic logging on those two. Fixed to $ENABLE_LOGGING. watchdog_orchestrator.sh and array_started.sh had no trailing exit, so their exit codes reflected whatever the last command happened to return rather than actual success/failure. transcode_management.sh had no failure notification and no summary at all. Also made transcode_management.sh's two-script pipeline config-driven (TRANSCODE_MANAGEMENT_SCRIPTS in master.conf) instead of hardcoded, for room to extend it later without editing the orchestrator itself.
This commit is contained in:
@@ -942,41 +942,40 @@ array_started.sh
|
||||
## ━━━ ADDING A NEW ORCHESTRATOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
If you find yourself running 3+ related scripts on the same schedule, wrap them
|
||||
in a new orchestrator. Model directly on `media_management.sh` which has the
|
||||
in a new orchestrator. Model directly on `daily_sync_maintenance.sh` which has the
|
||||
complete pattern — dry-run passthrough, status display, pass/fail tracking, summary.
|
||||
|
||||
Child-script execution goes through the shared `run_orch_child()` helper in
|
||||
`common.sh` — never hand-roll a per-file `run_job()` loop. It resolves the entry
|
||||
against `$ECOSYSTEM_ROOT`, threads `--dry-run`/`--log` from `$DRY_RUN`/`$ENABLE_LOGGING`
|
||||
automatically (never `$VERBOSE` — nothing in this codebase assigns it), and tracks
|
||||
into `JOB_PASS`/`JOB_FAIL` arrays the caller declares.
|
||||
|
||||
```bash
|
||||
# Minimal skeleton — the full pattern in its simplest form:
|
||||
#!/bin/bash
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
ECOSYSTEM_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
||||
source "$ECOSYSTEM_ROOT/load_config.sh"
|
||||
parse_args "$@"
|
||||
|
||||
SCRIPTS_ROOT="$SCRIPT_DIR/.."
|
||||
PASS=()
|
||||
FAIL=()
|
||||
JOB_PASS=()
|
||||
JOB_FAIL=()
|
||||
|
||||
# Read job list from master.conf — never hardcode jobs in the orchestrator
|
||||
for script_entry in "${MY_MAINTENANCE_JOBS[@]:-}"; do
|
||||
[[ -z "$script_entry" ]] && continue
|
||||
|
||||
read -r -a parts <<< "$script_entry"
|
||||
script_path="$SCRIPTS_ROOT/${parts[0]}"
|
||||
script_name=$(basename "${parts[0]}")
|
||||
extra_args=("${parts[@]:1}")
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && extra_args+=("--dry-run")
|
||||
|
||||
if bash "$script_path" "${extra_args[@]}"; then
|
||||
PASS+=("$script_name")
|
||||
else
|
||||
FAIL+=("$script_name")
|
||||
fi
|
||||
run_orch_child "$script_entry"
|
||||
done
|
||||
|
||||
# One summary — one notification
|
||||
echo "Passed: ${#PASS[@]} Failed: ${#FAIL[@]}"
|
||||
[[ ${#FAIL[@]} -gt 0 ]] && \
|
||||
notify "My maintenance failed on $(hostname) ($MY_ID) — ${FAIL[*]}" \
|
||||
# One summary — one notification. On a frequent (sub-daily) cadence, keep the
|
||||
# healthy path to a single line and reserve the full breakdown for failure/--log —
|
||||
# see watchdog_orchestrator.sh or transcode_management.sh for that split.
|
||||
echo "Passed: ${#JOB_PASS[@]} Failed: ${#JOB_FAIL[@]}"
|
||||
if [[ ${#JOB_FAIL[@]} -gt 0 ]]; then
|
||||
notify "My maintenance failed on $(hostname) ($MY_ID) — ${JOB_FAIL[*]}" \
|
||||
"My Orchestrator" "warning"
|
||||
exit 1
|
||||
fi
|
||||
exit 0
|
||||
```
|
||||
Reference in New Issue
Block a user