Bring script headers onto the template and close safeguard gaps
Headers claimed protections the code never had, and several destructive paths had no guard against a collapsed config value.
This commit is contained in:
@@ -55,13 +55,33 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — all launched scripts require root
|
||||
# acquire_lock — prevents duplicate array start launches
|
||||
# detect_hosts() — MY_ID in notifications
|
||||
# platform_require_cmd — notify validated before use
|
||||
# chmod +x auto-fix — non-executable scripts fixed before launch
|
||||
# Full path on failure — shows exact path for debugging
|
||||
# notify on failures — alert if any script fails to launch
|
||||
# Root Enforcement
|
||||
# Every script launched here requires root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents duplicate array start launches. Unraid can fire the array
|
||||
# start hook more than once, and a second pass would re-launch continuous scripts
|
||||
# that are already running.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID for notifications.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if ARRAY_START_SCRIPTS is empty. An empty
|
||||
# list would silently bring the array up with no ramdisk, no network setup, no
|
||||
# watchdogs and no fallback — while reporting a clean start.
|
||||
#
|
||||
# Executable Auto-Fix
|
||||
# Non-executable scripts are chmod +x'd before launch. A permission bit lost to a
|
||||
# git checkout or a file copy should not silently disable a boot-time component.
|
||||
#
|
||||
# Full Path on Failure
|
||||
# Failures report the exact resolved path, so a missing script is immediately
|
||||
# distinguishable from a script that ran and failed.
|
||||
#
|
||||
# Failure Notification
|
||||
# Any script that fails to launch raises a notification — array start is unattended,
|
||||
# so a silent failure here would only surface much later as a missing service.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -109,6 +129,16 @@ acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#ARRAY_START_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "ARRAY_START_SCRIPTS is empty — no array start scripts will run"
|
||||
error "Check ARRAY_START_SCRIPTS in master.conf"
|
||||
notify "array start scripts skipped on $(hostname) ($MY_ID) — ARRAY_START_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — scripts will not be launched"
|
||||
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -40,11 +40,28 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — all stop scripts require root
|
||||
# acquire_lock — prevents concurrent array stop runs
|
||||
# detect_hosts() — MY_ID in notifications and logs
|
||||
# platform_require_cmd — notify validated before use
|
||||
# notify on failures — alert if any stop script fails
|
||||
# Root Enforcement
|
||||
# Every stop script launched here requires root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents concurrent array stop runs. Two overlapping shutdown
|
||||
# sequences would fight over the same containers.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID for notifications and logs.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if ARRAY_STOP_SCRIPTS is empty. An empty
|
||||
# list means the array stops without saving the conf cache or gracefully stopping
|
||||
# containers — the failure would only be discovered at the next boot.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# A failing stop script is recorded and the remaining ones still run. Abandoning the
|
||||
# shutdown sequence partway would leave more state unsaved than continuing does.
|
||||
#
|
||||
# Failure Notification
|
||||
# Any failing stop script raises a notification. Shutdown is unattended and its
|
||||
# failures are invisible until they cause a problem on the way back up.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -97,6 +114,16 @@ fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#ARRAY_STOP_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "ARRAY_STOP_SCRIPTS is empty — no array stop scripts will run"
|
||||
error "Check ARRAY_STOP_SCRIPTS in master.conf"
|
||||
notify "array stop scripts skipped on $(hostname) ($MY_ID) — ARRAY_STOP_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no stop scripts will be executed"
|
||||
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -44,11 +44,31 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — rsync and container stop/start require root
|
||||
# acquire_lock "strict" — no pile-up; skip cycle if prior run still active
|
||||
# detect_hosts() — MY_ID and REMOTE_ID for routing and logs
|
||||
# resolve_remote_ip — confirms remote reachability before any transfer
|
||||
# RSYNC_ENABLED gate — global kill switch respected before any rsync call
|
||||
# Root Enforcement
|
||||
# rsync over SSH and container stop/start both require root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock in strict mode — a cycle is skipped rather than queued if the previous
|
||||
# one is still running. At a 30-minute cadence, queuing would let a slow sync stack
|
||||
# windows behind it.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID and REMOTE_ID for routing and logs.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if CRITICAL_MAINTENANCE_SCRIPTS is empty —
|
||||
# a silently empty critical tier would stop downloader resets and play-state sync
|
||||
# while still reporting success every 30 minutes.
|
||||
#
|
||||
# Remote IP Resolution
|
||||
# resolve_remote_ip confirms the partner is reachable before any transfer is attempted.
|
||||
#
|
||||
# RSYNC_ENABLED Gate
|
||||
# The global kill switch is respected before any rsync call, so disabling rsync
|
||||
# ecosystem-wide genuinely stops it here too.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# A failing job is recorded and the rest of the tier still runs.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -98,6 +118,16 @@ fi
|
||||
acquire_lock "strict"
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#CRITICAL_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "CRITICAL_MAINTENANCE_SCRIPTS is empty — no critical maintenance scripts will run"
|
||||
error "Check CRITICAL_MAINTENANCE_SCRIPTS in master.conf"
|
||||
notify "critical maintenance scripts skipped on $(hostname) ($MY_ID) — CRITICAL_MAINTENANCE_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
resolve_remote_ip
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
@@ -59,13 +59,40 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — rsync and docker operations require root
|
||||
# acquire_lock — prevents concurrent daily windows
|
||||
# detect_hosts() — aliases correct per-host share lists
|
||||
# check_connectivity — verified before any rsync
|
||||
# check_remote_rootfs — aborts rsync if remote rootfs nearly full
|
||||
# Non-fatal jobs — a failed job logs and continues; remaining jobs still run
|
||||
# notify on failure — successful daily run produces no notification
|
||||
# Root Enforcement
|
||||
# rsync and docker operations require root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents two daily windows overlapping — the window is long and a
|
||||
# second pass would contend for the same shares and containers.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() aliases the correct per-host share and script lists.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if DAILY_MAINTENANCE_SCRIPTS is empty. This
|
||||
# is the largest tier in the ecosystem — an empty list would silently skip git pull,
|
||||
# permissions, cleaners, arr cleanup and docker updates while reporting a clean run.
|
||||
#
|
||||
# Connectivity Check
|
||||
# check_connectivity is verified before any rsync is attempted.
|
||||
#
|
||||
# Remote Rootfs Check
|
||||
# check_remote_rootfs aborts rsync if the remote rootfs is nearly full, rather than
|
||||
# pushing data to a partner that cannot hold it.
|
||||
#
|
||||
# Drive Temperature Escalation
|
||||
# rsync.sh's exit code is honoured per share: exit 1 (temp WARN) skips that share and
|
||||
# continues; exit 2 (temp CRITICAL) sets ABORT_ALL_SYNCS so every remaining share in
|
||||
# the window is skipped and a notification is raised. Continuing to hammer drives that
|
||||
# are already too hot is how a thermal warning becomes a dead disk.
|
||||
#
|
||||
# Non-Fatal Jobs
|
||||
# A failed job is logged and the remaining jobs still run. Partial completion of a
|
||||
# maintenance window beats abandoning it at the first error.
|
||||
#
|
||||
# Quiet on Success
|
||||
# A successful daily run produces no notification — only failures surface.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -123,6 +150,16 @@ if ! command -v docker &>/dev/null; then
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#DAILY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "DAILY_MAINTENANCE_SCRIPTS is empty — no daily maintenance scripts will run"
|
||||
error "Check DAILY_MAINTENANCE_SCRIPTS in master.conf"
|
||||
notify "daily maintenance scripts skipped on $(hostname) ($MY_ID) — DAILY_MAINTENANCE_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
resolve_remote_ip
|
||||
|
||||
acquire_lock
|
||||
|
||||
@@ -40,13 +40,36 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — scripts called here require root
|
||||
# acquire_lock — prevents concurrent intermediate windows
|
||||
# detect_hosts() — aliases correct per-host share lists
|
||||
# check_connectivity — verified before any rsync (skipped if no shares)
|
||||
# check_remote_rootfs — aborts rsync if remote rootfs nearly full
|
||||
# Non-fatal jobs — a failed arr_sync warns but does not block rsync or artwork fetch
|
||||
# Minimal on success — runs 6x/day; full breakdown only on failure or --log
|
||||
# Root Enforcement
|
||||
# Every script called from here requires root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents concurrent intermediate windows.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() aliases the correct per-host share lists.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if INTERMEDIATE_MAINTENANCE_SCRIPTS is empty,
|
||||
# rather than running six no-op windows a day that all report success.
|
||||
#
|
||||
# Connectivity Check
|
||||
# check_connectivity is verified before any rsync, and skipped entirely when no shares
|
||||
# are configured — there is nothing to reach a partner for.
|
||||
#
|
||||
# Remote Rootfs Check
|
||||
# check_remote_rootfs aborts rsync if the remote rootfs is nearly full.
|
||||
#
|
||||
# Drive Temperature Escalation
|
||||
# rsync.sh's exit code is honoured per share: exit 1 skips that share, exit 2 aborts
|
||||
# every remaining sync in the window and notifies.
|
||||
#
|
||||
# Non-Fatal Jobs
|
||||
# A failing arr_sync warns but does not block rsync or the artwork fetch that follow it.
|
||||
#
|
||||
# Minimal on Success
|
||||
# Runs six times a day, so the full breakdown only prints on failure or with --log.
|
||||
# A quiet run is the normal outcome and should not fill the log.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -101,6 +124,16 @@ fi
|
||||
acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#INTERMEDIATE_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "INTERMEDIATE_MAINTENANCE_SCRIPTS is empty — no intermediate maintenance scripts will run"
|
||||
error "Check INTERMEDIATE_MAINTENANCE_SCRIPTS in master.conf"
|
||||
notify "intermediate maintenance scripts skipped on $(hostname) ($MY_ID) — INTERMEDIATE_MAINTENANCE_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
resolve_remote_ip
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
@@ -45,12 +45,34 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — ZFS scrub, SMART tests require root
|
||||
# acquire_lock — prevents concurrent monthly runs
|
||||
# detect_hosts() — MY_ID in notifications and logs
|
||||
# Uptime gate — MONTHLY_UPTIME_THRESHOLD_DAYS must be met
|
||||
# Interval gate — MONTHLY_RUN_INTERVAL_DAYS since last run must be met
|
||||
# --force flag — bypasses both gates for manual override
|
||||
# Root Enforcement
|
||||
# ZFS scrub and SMART tests require root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents concurrent monthly runs. These are long jobs — a scrub can run
|
||||
# for hours — and two at once would double the I/O cost for no benefit.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID for notifications and logs.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if MONTHLY_MAINTENANCE_SCRIPTS is empty. A
|
||||
# monthly job that silently does nothing is the hardest kind to notice missing.
|
||||
#
|
||||
# Uptime Gate
|
||||
# MONTHLY_UPTIME_THRESHOLD_DAYS must be met before the run proceeds. Heavy full-disk
|
||||
# work immediately after a boot competes with everything else still starting up.
|
||||
#
|
||||
# Interval Gate
|
||||
# MONTHLY_RUN_INTERVAL_DAYS since the last successful run must have elapsed. The
|
||||
# schedule fires more often than the work should actually happen, so the gate — not
|
||||
# the cron entry — is what defines the real cadence.
|
||||
#
|
||||
# Force Override
|
||||
# --force bypasses both gates for a deliberate manual run.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# A failing job is recorded and the remaining jobs still run.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -113,6 +135,16 @@ acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#MONTHLY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "MONTHLY_MAINTENANCE_SCRIPTS is empty — no monthly maintenance scripts will run"
|
||||
error "Check MONTHLY_MAINTENANCE_SCRIPTS in master.conf"
|
||||
notify "monthly maintenance scripts skipped on $(hostname) ($MY_ID) — MONTHLY_MAINTENANCE_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no scripts will be executed"
|
||||
[[ "$FORCE_RUN" == true ]] && warn "FORCE — uptime and interval gates bypassed"
|
||||
|
||||
|
||||
@@ -39,12 +39,33 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — required for the docker/system reads used in the report
|
||||
# acquire_lock — prevents overlapping weekly runs
|
||||
# detect_hosts() — MY_ID in banner and summary
|
||||
# Non-fatal steps — a failed script is logged; remaining scripts still run
|
||||
# Flag pass-through — --dry-run and --log forwarded to all child scripts
|
||||
# notify() on failure — pushed only outside --dry-run, matching the runtime-mode contract below
|
||||
# Root Enforcement
|
||||
# Required for the docker and system reads the report is built from.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents overlapping weekly runs.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID for the banner and summary.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if COFFEE_REPORT_SCRIPTS is empty. A report
|
||||
# that silently contains nothing still arrives looking like a report.
|
||||
#
|
||||
# Read-Only by Composition
|
||||
# Every child here is a reporting script. This orchestrator changes nothing itself —
|
||||
# it only sequences reads and assembles their output.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# A failing script is logged and the remaining ones still run, so one unavailable
|
||||
# subsystem costs a section of the report rather than the whole thing.
|
||||
#
|
||||
# Flag Pass-Through
|
||||
# --dry-run and --log are forwarded to every child script.
|
||||
#
|
||||
# Notification Contract
|
||||
# notify() fires on failure only outside --dry-run, matching the runtime-mode contract
|
||||
# below — a dry run never sends anything outward.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -91,6 +112,16 @@ acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#COFFEE_REPORT_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "COFFEE_REPORT_SCRIPTS is empty — no coffee report scripts will run"
|
||||
error "Check COFFEE_REPORT_SCRIPTS in master.conf"
|
||||
notify "coffee report scripts skipped on $(hostname) ($MY_ID) — COFFEE_REPORT_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Helpers ━━━
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -50,12 +50,37 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — mount and docker operations require root
|
||||
# acquire_lock — prevents concurrent 7-minute cycles overlapping
|
||||
# detect_hosts() — aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB per host
|
||||
# --dry-run — passed through to every script in TRANSCODE_MANAGEMENT_SCRIPTS
|
||||
# Exit code — worst exit code across all scripts returned to cron
|
||||
# notify() — pushed on failure, skipped in --dry-run
|
||||
# Root Enforcement
|
||||
# Mount and docker operations require root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents concurrent 7-minute cycles overlapping. Cleanup and the manager
|
||||
# both touch the same ramdisk, and two cycles at once could have one deleting files
|
||||
# while the other is measuring usage to decide whether to flip.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() aliases RAMDISK_PATH, TRANSCODE_SSD and RAMDISK_WARN_GB per host.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if TRANSCODE_MANAGEMENT_SCRIPTS is empty —
|
||||
# without it the ramdisk would silently stop being cleaned or flipped, and the first
|
||||
# symptom would be a full ramdisk stalling playback.
|
||||
#
|
||||
# Ordering Is Load-Bearing
|
||||
# Cleanup runs before the manager so the manager measures real active-session usage
|
||||
# rather than usage inflated by stale files. Reversing them would trigger flips that
|
||||
# a cleanup two seconds later would have made unnecessary.
|
||||
#
|
||||
# Dry Run Propagation
|
||||
# --dry-run is passed through to every script in TRANSCODE_MANAGEMENT_SCRIPTS.
|
||||
#
|
||||
# Any-Failure Exit Code
|
||||
# Exits 1 if any child failed, 0 otherwise — the individual exit codes are not
|
||||
# propagated, only whether anything failed. A failure in an early child is therefore
|
||||
# never masked by a later success.
|
||||
#
|
||||
# Notification Contract
|
||||
# notify() fires on failure and is skipped in --dry-run.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -112,6 +137,16 @@ fi
|
||||
# detect_hosts() sets MY_ID and aliases all HOST*_TRANSCODE_* vars
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#TRANSCODE_MANAGEMENT_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "TRANSCODE_MANAGEMENT_SCRIPTS is empty — no transcode management scripts will run"
|
||||
error "Check TRANSCODE_MANAGEMENT_SCRIPTS in master.conf"
|
||||
notify "transcode management scripts skipped on $(hostname) ($MY_ID) — TRANSCODE_MANAGEMENT_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — passing through to child scripts"
|
||||
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -49,11 +49,34 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — watchdog operations require root
|
||||
# acquire_lock — strict; no pile-up if prior cycle still active
|
||||
# detect_hosts() — MY_ID in logs and notifications
|
||||
# Array check — exits early if /mnt/user is not shfs-mounted
|
||||
# Startup grace — WATCHDOG_STARTUP_GRACE respected before any checks
|
||||
# Root Enforcement
|
||||
# Every watchdog launched from here requires root. Failing once at the top gives one
|
||||
# clear error instead of the same permission failure repeated per child.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock in strict mode — if the previous cycle is still running, this one exits
|
||||
# rather than queuing. At a one-minute cadence a waiting lock would pile up cycles
|
||||
# behind a slow watchdog and eventually run them all at once.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID for logs and notifications.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if WATCHDOG_ORCHESTRATOR_SCRIPTS is empty.
|
||||
# Without it the cycle reports "0/0 passed" and exits 0 every minute — indistinguishable
|
||||
# from a healthy run, while nothing at all is being monitored.
|
||||
#
|
||||
# Array Check
|
||||
# Exits early if /mnt/user is not shfs-mounted. Watchdogs that inspect shares would
|
||||
# otherwise read an unmounted array as missing data and act on it.
|
||||
#
|
||||
# Startup Grace
|
||||
# WATCHDOG_STARTUP_GRACE is respected before any checks run, so containers still
|
||||
# initialising after boot are not judged as unhealthy.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# run_orch_child() records a failing or missing watchdog and continues. One broken
|
||||
# watchdog never suppresses the rest of the chain.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -104,6 +127,16 @@ acquire_lock
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "WATCHDOG_ORCHESTRATOR_SCRIPTS is empty — no watchdogs will run"
|
||||
error "Check WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf"
|
||||
notify "watchdogs skipped on $(hostname) ($MY_ID) — WATCHDOG_ORCHESTRATOR_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
log "$ICON_GEAR Config: grace=${WATCHDOG_STARTUP_GRACE}s heartbeat=${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}/${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr scripts=${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
|
||||
log "$ICON_WATCHDOG Order: $(for s in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do printf '%s ' "${s##*/}"; done)"
|
||||
|
||||
|
||||
@@ -40,15 +40,35 @@
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — stop/start containers and rsync require root
|
||||
# acquire_lock — prevents concurrent weekly windows
|
||||
# detect_hosts() — MY_ID in banner, summary, and notifications
|
||||
# check_connectivity — verifies remote before any remote operations
|
||||
# check_remote_rootfs — aborts rsync if remote rootfs nearly full
|
||||
# DOCKER_TIMEOUT — all docker calls protected
|
||||
# SSH_TIMEOUT — all SSH calls protected
|
||||
# platform_require_cmd — notify validated before use
|
||||
# Silent on success — runs weekly; only failures warrant notification
|
||||
# Root Enforcement
|
||||
# Container stop/start and rsync both require root.
|
||||
#
|
||||
# Lock Acquisition
|
||||
# acquire_lock prevents concurrent weekly windows. This window stops Emby and the auth
|
||||
# stack — two overlapping runs would fight over the same critical containers.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() sets MY_ID for the banner, summary and notifications.
|
||||
#
|
||||
# Empty Job List Guard
|
||||
# Exits with an error and a notification if WEEKLY_MAINTENANCE_SCRIPTS is empty, rather
|
||||
# than taking the weekly outage window and doing nothing with it.
|
||||
#
|
||||
# Connectivity Check
|
||||
# check_connectivity verifies the remote before any remote operation is attempted.
|
||||
#
|
||||
# Remote Rootfs Check
|
||||
# check_remote_rootfs aborts rsync if the remote rootfs is nearly full.
|
||||
#
|
||||
# Timeout Protection
|
||||
# DOCKER_TIMEOUT bounds every docker call and SSH_TIMEOUT every SSH call, so neither a
|
||||
# hung daemon nor an unresponsive partner can hold the weekly window open indefinitely.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# A failing job is recorded and the remaining jobs still run.
|
||||
#
|
||||
# Silent on Success
|
||||
# Runs weekly; only failures warrant a notification.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
@@ -109,6 +129,16 @@ if ! command -v docker &>/dev/null; then
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# An unconfigured job list would run nothing and still report "0/0 passed" — indistinguishable
|
||||
# from a healthy run. Fail loudly instead of silently doing no work.
|
||||
if [[ ${#WEEKLY_MAINTENANCE_SCRIPTS[@]} -eq 0 ]]; then
|
||||
error "WEEKLY_MAINTENANCE_SCRIPTS is empty — no weekly maintenance scripts will run"
|
||||
error "Check WEEKLY_MAINTENANCE_SCRIPTS in master.conf"
|
||||
notify "weekly maintenance scripts skipped on $(hostname) ($MY_ID) — WEEKLY_MAINTENANCE_SCRIPTS is empty" \
|
||||
"$(basename "$0" .sh)" "warning"
|
||||
exit 1
|
||||
fi
|
||||
resolve_remote_ip
|
||||
|
||||
WINDOW_START=$(date +%s)
|
||||
|
||||
Reference in New Issue
Block a user