Add full banner headers to all scripts across the codebase
Every script now has the established header format: PURPOSE with ─────── separator, OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and RUNTIME MODES — structured with full ====== banner sections throughout. Orchestrators converted from compact ── inline format to full banners. Stale emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
This commit is contained in:
Gmer4Lfe
@@ -2,61 +2,91 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Array Start Orchestrator ===================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Single entry point for array start — fired by the Varaverk plugin's
|
||||
# disks_mounted event hook (Plugin/unraid/event/disks_mounted/array_start_jobs).
|
||||
# Launches everything configured in ARRAY_START_SCRIPTS in master.conf.
|
||||
# This script exits after launching all scripts — the event hook sees it complete normally.
|
||||
#
|
||||
# ── WHAT IT LAUNCHES ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Configured in master.conf ARRAY_START_SCRIPTS — no changes to this script ever needed.
|
||||
# Current order (order matters — see below):
|
||||
#
|
||||
# ONE-SHOT (run and exit naturally):
|
||||
# System_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
|
||||
# System_Essentials/inotify_tuning.sh — raise inotify limits before containers start
|
||||
# System_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
|
||||
# System_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
|
||||
# Transcodes/ramdisk_setup.sh — create tmpfs + symlink before Emby starts
|
||||
# System_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
|
||||
# System_Essentials/inotify_tuning.sh — raise inotify limits before containers start
|
||||
# System_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
|
||||
# System_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
|
||||
# Transcodes/ramdisk_setup.sh — create tmpfs + symlink before Emby starts
|
||||
# Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
|
||||
#
|
||||
# CONTINUOUS (run until array stops):
|
||||
# Fallback/fallback.sh — mutual fallback monitor
|
||||
# Fallback/fallback.sh — mutual fallback monitor
|
||||
#
|
||||
# NOTE: watchdogs (docker, system, stability) are NOT launched here.
|
||||
# They run via watchdog_orchestrator.sh every 15 min (cron), not as daemons.
|
||||
#
|
||||
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
|
||||
# unraid_api_key_renew.sh — before anything else — self-heals API registry at boot
|
||||
# inotify_tuning.sh — must run BEFORE Code-Server and other containers start
|
||||
# containers that start with low inotify limits keep them ✅
|
||||
# docker_syslog_filter — must run BEFORE any container starts creating veth interfaces
|
||||
# ramdisk_setup.sh — must run BEFORE Emby starts transcoding
|
||||
# docker_network_connect — must run BEFORE watchdogs check container states
|
||||
# fallback.sh — last — needs everything else stable to make decisions
|
||||
#
|
||||
# ── ONE-SHOT vs CONTINUOUS DETECTION ─────────────────────────────────────────────────────────
|
||||
# Script is launched in background with bash script.sh &
|
||||
# After 1 second: if PID still alive → continuous (running in background)
|
||||
# if PID dead + exit 0 → one-shot completed successfully
|
||||
# if PID dead + exit N → failure
|
||||
# if PID dead + exit 0 → one-shot completed successfully
|
||||
# if PID dead + exit N → failure
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — all launched scripts require root
|
||||
# acquire_lock — prevents duplicate array start launches
|
||||
# detect_hosts() — MY_ID in notifications
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Order Is Load-Bearing
|
||||
# inotify limits must be raised before containers start — containers that
|
||||
# start with low limits keep them. Ramdisk must exist before Emby starts.
|
||||
# Docker networks must be connected before watchdogs check container states.
|
||||
# fallback.sh goes last — it needs everything else stable to make decisions.
|
||||
#
|
||||
# Configuration Owns the List
|
||||
# ARRAY_START_SCRIPTS in master.conf is the only place scripts are added or
|
||||
# removed. This orchestrator never needs to be edited to change what runs —
|
||||
# one-shot vs continuous behaviour is auto-detected from the PID after launch.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — all launched scripts require root
|
||||
# acquire_lock — prevents duplicate array start launches
|
||||
# detect_hosts() — MY_ID in notifications
|
||||
# platform_require_cmd — notify validated before use
|
||||
# chmod +x auto-fix — non-executable scripts fixed before launch
|
||||
# Full path on failure — shows exact path for debugging
|
||||
# notify on failures — alert if any script fails to launch
|
||||
# chmod +x auto-fix — non-executable scripts fixed before launch
|
||||
# Full path on failure — shows exact path for debugging
|
||||
# notify on failures — alert if any script fails to launch
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# ARRAY_START_SCRIPTS — ordered list of scripts to launch at array start
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# array_started.sh — normal launch (called by Varaverk disks_mounted event hook)
|
||||
# array_started.sh --dry-run — show what would be launched without launching
|
||||
# array_started.sh --status — show configured scripts and their current state
|
||||
# array_started.sh --log — verbose output per script
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# array_started.sh
|
||||
# Normal launch — called by Varaverk disks_mounted event hook.
|
||||
#
|
||||
# array_started.sh --dry-run
|
||||
# Show what would be launched without launching.
|
||||
#
|
||||
# array_started.sh --status
|
||||
# Show configured scripts and their current state.
|
||||
#
|
||||
# array_started.sh --log
|
||||
# Verbose output per script.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,40 +2,74 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Array Stop Orchestrator ====================================
|
||||
# ==============================================================================================
|
||||
# Planned shutdown orchestrator — stops all active processes cleanly before array maintenance.
|
||||
# Runs ARRAY_STOP_SCRIPTS from master.conf sequentially, each confirmed complete before next.
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Planned shutdown orchestrator — stops all active processes cleanly before
|
||||
# array maintenance. Runs ARRAY_STOP_SCRIPTS from master.conf sequentially,
|
||||
# each confirmed complete before the next starts.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# 1. user_scripts_stop.sh — kill background user scripts (prevents new operations)
|
||||
# 2. rsync_stop.sh --rsync-only — kill rsync; skip container recovery (handled in step 4)
|
||||
# 3. mover_stop.sh — stop mover after rsync (both write to same paths)
|
||||
# 4. docker_container_stop.sh — stop all containers one-by-one with verification
|
||||
#
|
||||
# ── WHY THIS ORDER ────────────────────────────────────────────────────────────────────────────
|
||||
# User scripts stopped first — they can spawn new rsync/docker operations mid-shutdown.
|
||||
# Rsync before mover — both write to the same paths; running together risks corruption.
|
||||
# Containers last — apps should stay available as long as possible during shutdown prep.
|
||||
# Unlike array_started.sh, all scripts run in the foreground. Each must complete
|
||||
# (pass or fail) before the next starts — a failed stop is noted but does not
|
||||
# prevent remaining steps from running.
|
||||
#
|
||||
# ── SEQUENTIAL vs BACKGROUND ─────────────────────────────────────────────────────────────────
|
||||
# Unlike array_started.sh, all scripts run in the foreground. Each must complete (pass or fail)
|
||||
# before the next starts — a failed stop is noted but does not prevent remaining steps.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — all stop scripts require root
|
||||
# acquire_lock — prevents concurrent array stop runs
|
||||
# detect_hosts() — MY_ID in notifications and logs
|
||||
# Order Is Load-Bearing
|
||||
# User scripts are stopped first — they can spawn new rsync or docker operations
|
||||
# mid-shutdown. Rsync stops before mover — both write to the same paths and
|
||||
# running together risks corruption. Containers stop last — apps should stay
|
||||
# available as long as possible during shutdown prep.
|
||||
#
|
||||
# Non-Fatal Steps
|
||||
# A failed stop step is logged and notified but does not abort the sequence.
|
||||
# Remaining scripts still run — a partial stop is better than a halted one.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — all stop scripts require root
|
||||
# acquire_lock — prevents concurrent array stop runs
|
||||
# detect_hosts() — MY_ID in notifications and logs
|
||||
# platform_require_cmd — notify validated before use
|
||||
# Non-fatal steps — a failed step is logged but remaining steps still run
|
||||
# notify on failures — alert if any stop script fails
|
||||
# notify on failures — alert if any stop script fails
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# ARRAY_STOP_SCRIPTS — ordered list of stop scripts to run
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# array_stopping.sh — run full stop sequence
|
||||
# array_stopping.sh --dry-run — preview without stopping anything
|
||||
# array_stopping.sh --status — show configured scripts and exit
|
||||
# array_stopping.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# array_stopping.sh
|
||||
# Run full stop sequence.
|
||||
#
|
||||
# array_stopping.sh --dry-run
|
||||
# Preview without stopping anything.
|
||||
#
|
||||
# array_stopping.sh --status
|
||||
# Show configured scripts and exit.
|
||||
#
|
||||
# array_stopping.sh --log
|
||||
# Verbose output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,46 +2,81 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Critical Sync Maintenance ======================================
|
||||
# ==============================================================================================
|
||||
# Orchestrator for time-sensitive syncs that run every 30 minutes.
|
||||
# Keeps the mirror current between the less frequent daily and weekly windows.
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Orchestrator for time-sensitive syncs running every 30 minutes. Keeps the
|
||||
# mirror current between the less frequent daily and weekly windows.
|
||||
# Schedule: */30 * * * * (every 30 minutes via User Scripts plugin)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# 1. Critical-Data rsync — auth stack, NPM config, certs (containers stopped both sides)
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# 1. Critical-Data rsync — auth stack, NPM config, certs (containers stopped both sides)
|
||||
# 2. CRITICAL_MAINTENANCE_SCRIPTS — play_state_sync + any other per-window scripts
|
||||
# 3. partnership --check — read both state files, detect changes, act accordingly
|
||||
# 3. partnership --check — read both state files, detect changes, act accordingly
|
||||
#
|
||||
# ── WHY EVERY 30 MINUTES ──────────────────────────────────────────────────────────────────────
|
||||
# Auth stack changes (new users, proxy rules, certs) propagate within 30min ✅
|
||||
# Emby watch states stay in sync — mirror users see correct playback position ✅
|
||||
# Partnership state changes detected and acted on quickly ✅
|
||||
# Lock prevents: daily rsync doing Critical-Data mid-critical window ✅
|
||||
#
|
||||
# ── RSYNC GATE ────────────────────────────────────────────────────────────────────────────────
|
||||
# RSYNC GATE
|
||||
# RSYNC_ENABLED=false → skips all syncs (global gate)
|
||||
# CRITICAL_RSYNC_ENABLED=false → skips critical syncs only (per-orchestrator gate)
|
||||
# partnership --check always runs regardless — state check doesn't need rsync
|
||||
# partnership --check always runs regardless — state check doesn't need rsync.
|
||||
#
|
||||
# ── LOCK BEHAVIOUR ────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock "strict" — if previous 30min run still going, skip this cycle entirely
|
||||
# Critical-Data taking > 30min is a problem worth knowing about
|
||||
# Strict mode prevents pile-up without waiting — log and move on ✅
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
|
||||
# Runs 48 times per day — clean runs must produce zero output ✅
|
||||
# Only failures and notable events produce visible output
|
||||
# Silent When Healthy
|
||||
# Runs 48 times per day — clean runs must produce zero output. Only failures
|
||||
# and notable events produce visible output.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# CRITICAL_RSYNC_ENABLED — enable/disable rsync section
|
||||
# CRITICAL_SYNC_SHARES — shares synced every 30min (HOST*_CRITICAL_SYNC_SHARES)
|
||||
# CRITICAL_MAINTENANCE_SCRIPTS — scripts run in critical window (optional)
|
||||
# PARTNERSHIP_ENABLED — enable/disable partnership check
|
||||
# Auth-First Window
|
||||
# Auth stack changes (new users, proxy rules, certs) propagate within 30min.
|
||||
# Emby watch states stay in sync — mirror users see correct playback position.
|
||||
# Partnership state changes detected and acted on quickly.
|
||||
#
|
||||
# Strict Lock, Never Queue
|
||||
# acquire_lock "strict" — if the previous 30-min run is still going, skip
|
||||
# this cycle entirely. Critical-Data taking > 30min is a problem worth
|
||||
# knowing about. Strict mode prevents pile-up without waiting.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — rsync and container stop/start require root
|
||||
# acquire_lock "strict" — no pile-up; skip cycle if prior run still active
|
||||
# detect_hosts() — MY_ID and REMOTE_ID for routing and logs
|
||||
# resolve_remote_ip — confirms remote reachability before any transfer
|
||||
# RSYNC_ENABLED gate — global kill switch respected before any rsync call
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# CRITICAL_RSYNC_ENABLED — enable/disable rsync section
|
||||
# CRITICAL_SYNC_SHARES — shares synced every 30min (HOST*_CRITICAL_SYNC_SHARES)
|
||||
# CRITICAL_MAINTENANCE_SCRIPTS — scripts run in critical window (optional)
|
||||
# PARTNERSHIP_ENABLED — enable/disable partnership check
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# critical_sync_maintenance.sh
|
||||
# Normal run.
|
||||
#
|
||||
# critical_sync_maintenance.sh --dry-run
|
||||
# Preview syncs without transferring.
|
||||
#
|
||||
# critical_sync_maintenance.sh --log
|
||||
# Verbose per-share output.
|
||||
#
|
||||
# critical_sync_maintenance.sh --status
|
||||
# Show configuration and exit.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# critical_sync_maintenance.sh — normal run
|
||||
# critical_sync_maintenance.sh --dry-run — preview syncs without transferring
|
||||
# critical_sync_maintenance.sh --log — verbose per-share output
|
||||
# critical_sync_maintenance.sh --status — show configuration and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,68 +2,101 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Daily Sync Maintenance =========================================
|
||||
# ==============================================================================================
|
||||
# Daily orchestrator — runs the full daily maintenance window in the correct order.
|
||||
# Schedule: 0 1 * * * (1am daily via User Scripts plugin)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# Pre-sync:
|
||||
# git_pull_execute.sh — pull latest scripts first, always
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Daily maintenance window orchestrator — runs the full daily sequence in the
|
||||
# correct order. Schedule: 0 1 * * * (1am daily via User Scripts plugin)
|
||||
#
|
||||
# Arr Sync (arr_sync.sh):
|
||||
# Syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
|
||||
# All nodes agree on tracked library before any files are transferred.
|
||||
# Remote nodes that don't have an arr running are skipped gracefully.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Rsync window (DAILY_SYNC_SHARES per host):
|
||||
# HOST*_DAILY_SYNC_SHARES — media shares spread to all nodes (no --delete)
|
||||
# HOST*_PERSONAL_SHARES — encrypted personal shares
|
||||
# Pre-sync:
|
||||
# git_pull_execute.sh — pull latest scripts first, always
|
||||
#
|
||||
# Post-sync maintenance (DAILY_MAINTENANCE_SCRIPTS):
|
||||
# media_shares_permissions.sh — fix ownership before arr cleanup
|
||||
# media_cleaner.sh anime — remove junk from anime shares
|
||||
# media_cleaner.sh media — remove junk from media shares
|
||||
# lidarr_cleanup.sh — remove orphaned music files (local arr = truth)
|
||||
# sonarr_cleanup.sh — remove orphaned TV files (local arr = truth)
|
||||
# radarr_cleanup.sh — remove orphaned movie files (local arr = truth)
|
||||
# docker_daily_restart.sh — restart containers needing daily restart
|
||||
# Arr Sync (arr_sync.sh):
|
||||
# Syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
|
||||
# All nodes agree on tracked library before any files are transferred.
|
||||
# Remote nodes that don't have an arr running are skipped gracefully.
|
||||
#
|
||||
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
|
||||
# git pull first — maintenance runs on latest code, not yesterday's
|
||||
# arr sync before rsync — all nodes track the same library before files are spread;
|
||||
# prevents remote arrs from searching for content already owned
|
||||
# rsync before cleanup — cleanup sees fully spread state, rsync has no --delete
|
||||
# permissions before arr cleanup — arrs need correct ownership to delete/rename
|
||||
# arr cleanup after permissions — clean ownership = successful orphan deletion
|
||||
# docker restart last — containers already processed by cleanup
|
||||
# Rsync window (DAILY_SYNC_SHARES per host):
|
||||
# HOST*_DAILY_SYNC_SHARES — media shares spread to all nodes (no --delete)
|
||||
# HOST*_PERSONAL_SHARES — encrypted personal shares
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# Bidirectional — same script runs on both servers, correct direction automatic.
|
||||
# detect_hosts() aliases DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars.
|
||||
# No manual HOST1/HOST2 comparisons — MY_ID routes correctly on any server.
|
||||
# Post-sync maintenance (DAILY_MAINTENANCE_SCRIPTS):
|
||||
# media_shares_permissions.sh — fix ownership before arr cleanup
|
||||
# media_cleaner.sh anime/media — remove junk from media and anime shares
|
||||
# lidarr/sonarr/radarr_cleanup.sh — remove orphaned files (local arr = truth)
|
||||
# docker_daily_restart.sh — restart containers needing daily restart
|
||||
#
|
||||
# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
|
||||
# rsync.sh returns exit codes for temperature issues:
|
||||
# exit 1 = temp WARN — skip this share, continue to next
|
||||
# exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
|
||||
# All other failures — skip share, continue to next
|
||||
# DRIVE TEMP HANDLING (rsync.sh exit codes):
|
||||
# exit 1 = temp WARN → skip this share, continue to next
|
||||
# exit 2 = temp CRITICAL → abort ALL remaining syncs in this window
|
||||
#
|
||||
# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
|
||||
# Runs daily at 1am — clean run should produce minimal output.
|
||||
# Each job reports log() on success (silent), warn()/error() on failure (visible).
|
||||
# Summary always shown — gives window timing and share/job counts.
|
||||
# Notify only on failure — successful daily maintenance doesn't need notification.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
|
||||
# HOST*_DAILY_SYNC_SHARES — shares pushed to mirror each day
|
||||
# HOST*_PERSONAL_SHARES — encrypted personal shares
|
||||
# DAILY_MAINTENANCE_SCRIPTS — maintenance jobs (permissions, cleanup, restart)
|
||||
# DAILY_RSYNC_ENABLED — enable/disable rsync section
|
||||
# Order Is Load-Bearing
|
||||
# git pull first — maintenance runs on latest code, not yesterday's. Arr sync
|
||||
# before rsync — all nodes track the same library before files are spread,
|
||||
# preventing remote arrs from searching for content already owned. Rsync before
|
||||
# cleanup — cleanup sees fully spread state. Permissions before arr cleanup —
|
||||
# arrs need correct ownership to delete/rename. Docker restart last — containers
|
||||
# already processed by cleanup.
|
||||
#
|
||||
# Host-Aware Routing
|
||||
# Bidirectional — same script runs on both servers, correct direction automatic.
|
||||
# detect_hosts() aliases DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars.
|
||||
# No manual HOST1/HOST2 comparisons needed.
|
||||
#
|
||||
# Silent When Healthy
|
||||
# Runs daily at 1am — clean runs produce minimal output. Each job logs silently
|
||||
# on success; failures surface to warn()/error(). Notify only on failure.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — rsync and docker operations require root
|
||||
# acquire_lock — prevents concurrent daily windows
|
||||
# detect_hosts() — aliases correct per-host share lists
|
||||
# check_connectivity — verified before any rsync
|
||||
# check_remote_rootfs — aborts rsync if remote rootfs nearly full
|
||||
# Non-fatal jobs — a failed job logs and continues; remaining jobs still run
|
||||
# notify on failure — successful daily run produces no notification
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_SYNC_SHARES — shares pushed to mirror each day
|
||||
# HOST*_PERSONAL_SHARES — encrypted personal shares
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DAILY_MAINTENANCE_SCRIPTS — maintenance jobs (permissions, cleanup, restart)
|
||||
# DAILY_RSYNC_ENABLED — enable/disable rsync section
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# daily_sync_maintenance.sh
|
||||
# Normal run.
|
||||
#
|
||||
# daily_sync_maintenance.sh --dry-run
|
||||
# Preview without syncing or changing.
|
||||
#
|
||||
# daily_sync_maintenance.sh --log
|
||||
# Verbose per-share/per-job output.
|
||||
#
|
||||
# daily_sync_maintenance.sh --status
|
||||
# Show configured shares and jobs.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# daily_sync_maintenance.sh — normal run
|
||||
# daily_sync_maintenance.sh --dry-run — preview without syncing or changing
|
||||
# daily_sync_maintenance.sh --log — verbose per-share/per-job output
|
||||
# daily_sync_maintenance.sh --status — show configured shares and jobs
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,56 +2,86 @@
|
||||
# ==============================================================================================
|
||||
# =========================== Intermediate Sync Maintenance ====================================
|
||||
# ==============================================================================================
|
||||
# 4-hour orchestrator — arr library reconciliation, artwork fetching, and optional rsync.
|
||||
# Schedule: 0 */4 * * * (every 4 hours)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.cache/vv/d/)
|
||||
# 2. arr_sync.sh — sync Lidarr/Sonarr/Radarr libraries across all nodes
|
||||
# 3. Rsync window (optional) — INTERMEDIATE_SYNC_SHARES, if any configured
|
||||
# 4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 4-hour orchestrator — arr library reconciliation, artwork fetching, and
|
||||
# optional rsync. Schedule: 0 */4 * * * (every 4 hours)
|
||||
#
|
||||
# ── WHY A SEPARATE ORCHESTRATOR ───────────────────────────────────────────────────────────────
|
||||
# arr libraries need to converge more frequently than once a day. If a remote node adds
|
||||
# something at 2am, the next daily window is 23 hours away — remote arrs search for content
|
||||
# they don't know is already owned. Running every 4 hours closes that gap.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# lidarr_missing_art.sh is idempotent — skips existing files, runs fast after initial fill.
|
||||
# Pairing it here means artwork catches up within 4 hours of a new album landing.
|
||||
# 1. conf_sync.sh --pull-only — refresh partner conf cache in RAM (/tmp/.cache/vv/d/)
|
||||
# 2. arr_sync.sh — sync Lidarr/Sonarr/Radarr libraries across all nodes
|
||||
# 3. Rsync window (optional) — INTERMEDIATE_SYNC_SHARES, if any configured
|
||||
# 4. INTERMEDIATE_MAINTENANCE_SCRIPTS — artwork fetch and any future 4-hour jobs
|
||||
#
|
||||
# Rsync is optional — INTERMEDIATE_SYNC_SHARES empty by default. Add shares to the config
|
||||
# if a subset of data needs mid-day propagation (e.g. watch state, metadata). Full media
|
||||
# share sync stays in the daily window.
|
||||
# DRIVE TEMP HANDLING (rsync.sh exit codes):
|
||||
# exit 1 = temp WARN → skip this share, continue to next
|
||||
# exit 2 = temp CRITICAL → abort ALL remaining syncs in this window
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_INTERMEDIATE_SYNC_SHARES → INTERMEDIATE_SYNC_SHARES.
|
||||
# Each server can have a different set of mid-day shares — configure in host*.conf.
|
||||
# Each script in INTERMEDIATE_MAINTENANCE_SCRIPTS handles its own host logic.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
|
||||
# Same as daily_sync_maintenance.sh:
|
||||
# exit 1 = temp WARN — skip this share, continue to next
|
||||
# exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
|
||||
# Closes the Library Gap
|
||||
# Arr libraries need to converge more frequently than once a day. If a remote
|
||||
# node adds something at 2am, the next daily window is 23 hours away — remote
|
||||
# arrs search for content they don't know is already owned. Running every 4
|
||||
# hours closes that gap.
|
||||
#
|
||||
# Idempotent Artwork
|
||||
# lidarr_missing_art.sh skips existing files and runs fast after initial fill.
|
||||
# Pairing it here means artwork catches up within 4 hours of a new album landing.
|
||||
#
|
||||
# Optional Rsync Layer
|
||||
# INTERMEDIATE_SYNC_SHARES is empty by default — the rsync step is skipped
|
||||
# entirely when nothing is configured. Add shares only if a subset of data
|
||||
# needs mid-day propagation. Full media share sync stays in the daily window.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — scripts called here require root
|
||||
# acquire_lock — prevents concurrent intermediate windows
|
||||
# detect_hosts() — aliases correct per-host share lists
|
||||
# check_connectivity — verified before any rsync (skipped if no shares)
|
||||
# check_remote_rootfs — aborts rsync if remote rootfs nearly full
|
||||
# Non-fatal jobs — a failed arr_sync warns but does not block rsync or artwork fetch
|
||||
# Silent on success — runs 4x/day, only failures warrant notification
|
||||
#
|
||||
# ── CONFIGURATION ─────────────────────────────────────────────────────────────────────────────
|
||||
# host*.conf: HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
|
||||
# master.conf: INTERMEDIATE_RSYNC_ENABLED — enable/disable rsync section (default: true)
|
||||
# master.conf: INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
|
||||
# master.conf: ARR_SYNC_ENABLED — toggle inside arr_sync.sh
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_INTERMEDIATE_SYNC_SHARES — shares synced mid-day (empty = rsync skipped)
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# INTERMEDIATE_RSYNC_ENABLED — enable/disable rsync section (default: true)
|
||||
# INTERMEDIATE_MAINTENANCE_SCRIPTS — jobs run after rsync
|
||||
# ARR_SYNC_ENABLED — toggle inside arr_sync.sh
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# intermediate_sync_maintenance.sh
|
||||
# Normal run.
|
||||
#
|
||||
# intermediate_sync_maintenance.sh --dry-run
|
||||
# Preview without changes.
|
||||
#
|
||||
# intermediate_sync_maintenance.sh --log
|
||||
# Verbose per-job output.
|
||||
#
|
||||
# intermediate_sync_maintenance.sh --status
|
||||
# Show configured shares/jobs and exit.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# intermediate_sync_maintenance.sh — normal run
|
||||
# intermediate_sync_maintenance.sh --dry-run — preview without changes
|
||||
# intermediate_sync_maintenance.sh --log — verbose per-job output
|
||||
# intermediate_sync_maintenance.sh --status — show configured shares/jobs and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,39 +2,86 @@
|
||||
# ==============================================================================================
|
||||
# ========================= Monthly Maintenance Orchestrator ===================================
|
||||
# ==============================================================================================
|
||||
# Uptime-triggered monthly maintenance — runs heavy tasks that need a stable, settled system.
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Uptime-triggered monthly maintenance — runs heavy tasks that need a stable,
|
||||
# settled system. Schedule: 0 0 15 * * (15th of each month at midnight)
|
||||
# Fires only when BOTH gates pass:
|
||||
# 1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS days
|
||||
# 2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS days ago (or never run)
|
||||
#
|
||||
# ── WHY UPTIME-GATED ─────────────────────────────────────────────────────────────────────────
|
||||
# A scheduled reboot resets uptime. Monthly tasks (ZFS scrub, SMART long test) need a
|
||||
# stable, settled system — not one that just rebooted. Uptime-gating ensures maintenance
|
||||
# only runs after the server has been healthy for a full month, never immediately post-boot.
|
||||
# If uptime or interval gate is not met on the 15th, the run is skipped until next month.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── HOW TO CALL ──────────────────────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 0 15 * * (15th of each month at midnight)
|
||||
# Silent exit 0 when either gate is not met. Only outputs when maintenance actually fires.
|
||||
# Runs MONTHLY_MAINTENANCE_SCRIPTS sequentially when both gates pass.
|
||||
# Silent exit 0 when either gate is not met — only outputs when maintenance fires.
|
||||
# If uptime or interval gate is not met on the 15th, the run is skipped until
|
||||
# next month.
|
||||
#
|
||||
# ── STATE FILE ────────────────────────────────────────────────────────────────────────────────
|
||||
# MONTHLY_LAST_RUN_FILE — /boot/config — survives reboots, available before array starts.
|
||||
# Written after each run (pass or partial fail). Format: Unix timestamp.
|
||||
# A reboot does NOT reset the last-run state — the interval gate survives independently
|
||||
# of the uptime gate. Both must pass before maintenance fires again.
|
||||
# STATE FILE
|
||||
# MONTHLY_LAST_RUN_FILE lives on /boot/config — survives reboots, available
|
||||
# before the array starts. Written after each run (pass or partial fail).
|
||||
# Format: Unix timestamp. A reboot does NOT reset the last-run state — the
|
||||
# interval gate survives independently of the uptime gate.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Uptime Gate Ensures Stability
|
||||
# A scheduled reboot resets uptime. Monthly tasks (ZFS scrub, SMART long test)
|
||||
# need a stable, settled system — not one that just rebooted. Both gates must
|
||||
# pass before maintenance fires, ensuring the server has been healthy for a
|
||||
# full month.
|
||||
#
|
||||
# State Survives Reboots
|
||||
# MONTHLY_LAST_RUN_FILE is on /boot/config (USB flash), not on the array.
|
||||
# It is always available regardless of array state, so the interval gate is
|
||||
# never lost to a reboot.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — ZFS scrub, SMART tests require root
|
||||
# acquire_lock — prevents concurrent monthly runs
|
||||
# detect_hosts() — MY_ID in notifications and logs
|
||||
# Uptime gate — MONTHLY_UPTIME_THRESHOLD_DAYS must be met
|
||||
# Interval gate — MONTHLY_RUN_INTERVAL_DAYS since last run must be met
|
||||
# --force flag — bypasses both gates for manual override
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# MONTHLY_MAINTENANCE_SCRIPTS — ordered list of scripts to run
|
||||
# MONTHLY_UPTIME_THRESHOLD_DAYS — minimum uptime in days before maintenance fires
|
||||
# MONTHLY_RUN_INTERVAL_DAYS — minimum days since last run before running again
|
||||
# MONTHLY_LAST_RUN_FILE — state file path — /boot/config, survives reboots
|
||||
# MONTHLY_LAST_RUN_FILE — state file path (/boot/config — survives reboots)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# monthly_maintenance.sh
|
||||
# Normal run — uptime + interval gates enforced.
|
||||
#
|
||||
# monthly_maintenance.sh --dry-run
|
||||
# Preview gate state and scripts without running.
|
||||
#
|
||||
# monthly_maintenance.sh --status
|
||||
# Show gate state, last run, and configured scripts.
|
||||
#
|
||||
# monthly_maintenance.sh --force
|
||||
# Bypass uptime + interval gates (manual override).
|
||||
#
|
||||
# monthly_maintenance.sh --log
|
||||
# Verbose output.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# monthly_maintenance.sh — normal run (uptime + interval gates enforced)
|
||||
# monthly_maintenance.sh --dry-run — preview gate state and scripts without running
|
||||
# monthly_maintenance.sh --status — show gate state, last run, and configured scripts
|
||||
# monthly_maintenance.sh --force — bypass uptime + interval gates (manual override)
|
||||
# monthly_maintenance.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,24 +2,71 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Sunday Morning Coffee Report ===================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Weekly monitoring orchestrator — runs all Sunday monitor scripts in sequence.
|
||||
# Designed to be read over coffee Sunday morning while the system is fully caught up
|
||||
# from the 2:30am maintenance window.
|
||||
# Schedule: 0 7 * * 0 (7am Sunday — after weekly_sync_maintenance.sh finishes at ~3am)
|
||||
# Designed to be read over coffee while the system is fully caught up from the
|
||||
# 2:30am maintenance window.
|
||||
# Schedule: 0 7 * * 0 (7am Sunday — after weekly_sync_maintenance.sh finishes)
|
||||
#
|
||||
# ── SCRIPTS (master.conf COFFEE_REPORT_SCRIPTS) ───────────────────────────────────────────────
|
||||
# Each script runs independently, logs to its own output, and notifies on findings.
|
||||
# All scripts receive --dry-run and --log flags from this orchestrator when set.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID — used in banner and summary.
|
||||
# HOST1 primary: runs all scripts. HOST2: limited to host-aware scripts only.
|
||||
# Runs COFFEE_REPORT_SCRIPTS from master.conf in order. Each script runs
|
||||
# independently, produces its own output, and notifies on findings.
|
||||
# All scripts receive --dry-run and --log flags from this orchestrator when set.
|
||||
#
|
||||
# HOST1 primary: runs all scripts.
|
||||
# HOST2: limited to host-aware scripts only — each script handles its own host logic.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Independent, Isolated Scripts
|
||||
# Each monitor script is fully self-contained — a failure in one does not
|
||||
# prevent the others from running. The orchestrator logs the failure and
|
||||
# continues to the next script.
|
||||
#
|
||||
# Runs After the Full Weekly Window
|
||||
# Scheduled 4+ hours after weekly_sync_maintenance.sh — the system is fully
|
||||
# synced and containers are back up before any monitoring reads run. Reports
|
||||
# reflect the settled post-maintenance state.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# detect_hosts() — MY_ID in banner and summary
|
||||
# Non-fatal steps — a failed script is logged; remaining scripts still run
|
||||
# Flag pass-through — --dry-run and --log forwarded to all child scripts
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# COFFEE_REPORT_SCRIPTS — ordered list of monitor scripts to run
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# sunday_morning_coffee_report.sh
|
||||
# Normal run.
|
||||
#
|
||||
# sunday_morning_coffee_report.sh --dry-run
|
||||
# Preview without any writes or notifications.
|
||||
#
|
||||
# sunday_morning_coffee_report.sh --log
|
||||
# Verbose per-script output.
|
||||
#
|
||||
# sunday_morning_coffee_report.sh --status
|
||||
# Show configured scripts and exit.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# sunday_morning_coffee_report.sh — normal run
|
||||
# sunday_morning_coffee_report.sh --dry-run — preview without any writes or notifications
|
||||
# sunday_morning_coffee_report.sh --log — verbose per-script output
|
||||
# sunday_morning_coffee_report.sh --status — show configured scripts and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,55 +2,82 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Transcode Management ===========================================
|
||||
# ==============================================================================================
|
||||
# Orchestrator — runs transcode_cleanup.sh then transcode_manager.sh in the correct order.
|
||||
# Replace individual transcode_manager and transcode_cleanup cron entries with this.
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Runs transcode_cleanup.sh then transcode_manager.sh in the correct order.
|
||||
# Replaces individual cron entries for each — this is the single cron entry.
|
||||
# Schedule: */7 * * * * (every 7 minutes via User Scripts plugin)
|
||||
#
|
||||
# ── WHY CLEANUP BEFORE MANAGER ────────────────────────────────────────────────────────────────
|
||||
# Cleanup runs first — removes stale segment files from ended sessions.
|
||||
# Manager runs after — threshold decisions based on real current usage post-cleanup.
|
||||
# Without this order, stale files inflate the ramdisk usage reading and trigger
|
||||
# unnecessary SSD flips even when active sessions would fit on the ramdisk.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── WHAT EACH SCRIPT DOES ─────────────────────────────────────────────────────────────────────
|
||||
# transcode_cleanup.sh — removes aged segment files not open by any process
|
||||
# uses lsof for O(1) per-file active check (never per-file lsof)
|
||||
# also triggers flip-back to ramdisk after cleanup if recovered ✅
|
||||
# transcode_cleanup.sh
|
||||
# Removes aged segment files not open by any process. Uses lsof for O(1)
|
||||
# per-file active check. Triggers flip-back to ramdisk after cleanup if
|
||||
# the ramdisk usage has recovered.
|
||||
#
|
||||
# transcode_manager.sh — checks ramdisk usage against thresholds
|
||||
# flips symlink between ramdisk and SSD as needed
|
||||
# writes one entry to TRANSCODE_DAILY_LOG after each run
|
||||
# shows active Emby sessions with play method
|
||||
# transcode_manager.sh
|
||||
# Checks ramdisk usage against thresholds. Flips the symlink between ramdisk
|
||||
# and SSD as needed. Writes one entry to TRANSCODE_DAILY_LOG after each run.
|
||||
# Shows active Emby sessions with play method.
|
||||
#
|
||||
# ── DAILY LOG ─────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_manager.sh writes to TRANSCODE_DAILY_LOG after each run:
|
||||
# DAILY LOG (written by transcode_manager.sh, not this orchestrator):
|
||||
# Format: DATE|RAMDISK_USED_GB|FLIP_COUNT|RAM_SESSIONS|SSD_SESSIONS
|
||||
# This orchestrator does NOT write its own log — manager handles it ✅
|
||||
# Log trimmed to TRANSCODE_LOG_RETENTION days by manager on each write.
|
||||
# Read by sunday_morning_coffee_report.sh and weekly_health_digest.sh.
|
||||
# Trimmed to TRANSCODE_LOG_RETENTION days on each write.
|
||||
# Read by sunday_morning_coffee_report.sh and weekly_health_digest.sh.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB etc.
|
||||
# Each server manages its own transcode location independently.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Cleanup First, Decide Later
|
||||
# Stale segment files from ended sessions inflate the ramdisk usage reading
|
||||
# and trigger unnecessary SSD flips even when active sessions would fit on
|
||||
# the ramdisk. Cleanup runs first so the manager measures real current usage.
|
||||
#
|
||||
# Delegated Logging
|
||||
# This orchestrator does not write its own log — transcode_manager.sh owns
|
||||
# the TRANSCODE_DAILY_LOG write. One log writer, one format, no duplication.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — mount and docker operations require root
|
||||
# acquire_lock — prevents concurrent 3-minute cycles overlapping
|
||||
# detect_hosts() — correct paths per host
|
||||
# acquire_lock — prevents concurrent 7-minute cycles overlapping
|
||||
# detect_hosts() — aliases RAMDISK_PATH, TRANSCODE_SSD, RAMDISK_WARN_GB per host
|
||||
# --dry-run — passed through to both child scripts
|
||||
# Exit code — worst exit code of both scripts returned
|
||||
# Exit code — worst exit code of both scripts returned to cron
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# TRANSCODE_DAILY_LOG — daily stats log (written by transcode_manager.sh)
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# TRANSCODE_DAILY_LOG — daily stats log (written by transcode_manager.sh)
|
||||
# TRANSCODE_LOG_RETENTION — days to keep (trimmed by manager)
|
||||
# TRANSCODE_STATE_FILE — current state (ramdisk_used, flip_count etc.)
|
||||
# All TRANSCODE_* threshold vars — see master.conf Transcode Manager section
|
||||
# TRANSCODE_STATE_FILE — current state (ramdisk_used, flip_count, etc.)
|
||||
# TRANSCODE_* threshold vars — see master.conf Transcode Manager section
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# transcode_management.sh
|
||||
# Normal run (every 7 minutes via cron).
|
||||
#
|
||||
# transcode_management.sh --dry-run
|
||||
# Preview without changes (passed to both child scripts).
|
||||
#
|
||||
# transcode_management.sh --status
|
||||
# Show configuration and current state.
|
||||
#
|
||||
# transcode_management.sh --log
|
||||
# Verbose output from both child scripts.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# transcode_management.sh — normal run (every 7 minutes via cron)
|
||||
# transcode_management.sh --dry-run — preview without changes (passed to children)
|
||||
# transcode_management.sh --status — show configuration and current state
|
||||
# transcode_management.sh --log — verbose output from both child scripts
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,50 +2,86 @@
|
||||
# ==============================================================================================
|
||||
# ============================ Watchdog Orchestrator ===========================================
|
||||
# ==============================================================================================
|
||||
# Runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle.
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Runs WATCHDOG_ORCHESTRATOR_SCRIPTS in order each cron cycle. Replaces the
|
||||
# continuous loops previously embedded in individual watchdog scripts — those
|
||||
# are now single-pass; this orchestrator provides the cadence.
|
||||
# Schedule: */15 * * * * (every 15 minutes)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# Driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf — add, remove, or reorder there.
|
||||
# Default: resource_watchdog → docker_watchdog → system_watchdog → unraid_api_key_renew → stability_watchdog
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
|
||||
# Resource Watchdog first — frees RAM and CPU before healing attempts container restarts.
|
||||
# Containers restarted into a resource-pressured system just fail again.
|
||||
# Docker Watchdog second — restarts with pressure already reduced, more likely to stabilise.
|
||||
# System Watchdog third — system component health after containers are healed.
|
||||
# API key renew fourth — self-heals unraid-api registry loss; check-first, silent when valid.
|
||||
# Stability Watchdog last — only reboots when all prior layers could not resolve the issue.
|
||||
# Order driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf.
|
||||
# Default: resource_watchdog → docker_watchdog → system_watchdog →
|
||||
# unraid_api_key_renew → stability_watchdog
|
||||
#
|
||||
# ── ARRAY CHECK ───────────────────────────────────────────────────────────────────────────────
|
||||
# Exits immediately if /mnt/user is not mounted as shfs (array not started).
|
||||
# Watchdogs check Docker containers and storage — meaningless without the array.
|
||||
# Prevents false positives and unnecessary reboots when array is stopped or stopping.
|
||||
# ARRAY CHECK
|
||||
# Exits immediately if /mnt/user is not mounted as shfs. Watchdogs check
|
||||
# Docker containers and storage — meaningless without the array. Prevents
|
||||
# false positives and unnecessary reboots when array is stopped or stopping.
|
||||
#
|
||||
# ── STARTUP GRACE ─────────────────────────────────────────────────────────────────────────────
|
||||
# No action until system uptime >= WATCHDOG_STARTUP_GRACE seconds.
|
||||
# Prevents false positives from containers still starting at array launch.
|
||||
# Each sub-script enforces this independently — orchestrator exits early to avoid log noise.
|
||||
# STARTUP GRACE
|
||||
# No action until system uptime >= WATCHDOG_STARTUP_GRACE seconds. Prevents
|
||||
# false positives from containers still starting at array launch. Each
|
||||
# sub-script enforces this independently — orchestrator exits early to avoid
|
||||
# log noise.
|
||||
#
|
||||
# ── OVERLAP PROTECTION ────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock() — exits immediately if a prior cycle is still in progress.
|
||||
# Prevents pile-up when a cycle runs long (daemon restart attempt = 30s, etc.).
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── REPLACES ──────────────────────────────────────────────────────────────────────────────────
|
||||
# Continuous loops previously in system_watchdog.sh and docker_watchdog.sh.
|
||||
# Those scripts are now single-pass — this orchestrator provides the cadence.
|
||||
# Pressure Before Healing
|
||||
# Resource Watchdog runs first — it frees RAM and CPU before any container
|
||||
# restart is attempted. Containers restarted into a resource-pressured system
|
||||
# just fail again. Docker Watchdog restarts with pressure already reduced.
|
||||
# System Watchdog checks component health after containers are healed.
|
||||
# Stability Watchdog reboots only when all prior layers could not resolve the
|
||||
# issue. API key renew is check-first and silent when valid.
|
||||
#
|
||||
# Never Queue
|
||||
# acquire_lock exits immediately if a prior cycle is still running. Prevents
|
||||
# pile-up when a cycle runs long (daemon restart attempt = 30s, etc.).
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — watchdog operations require root
|
||||
# acquire_lock — strict; no pile-up if prior cycle still active
|
||||
# detect_hosts() — MY_ID in logs and notifications
|
||||
# Array check — exits early if /mnt/user is not shfs-mounted
|
||||
# Startup grace — WATCHDOG_STARTUP_GRACE respected before any checks
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# WATCHDOG_ORCHESTRATOR_SCRIPTS — watchdogs to run, in order
|
||||
# WATCHDOG_STARTUP_GRACE — seconds after boot before checks activate
|
||||
# WATCHDOG_ORCHESTRATOR_HEARTBEAT — periodic heartbeat log toggle
|
||||
# WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS — heartbeat interval in hours
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# watchdog_orchestrator.sh — normal run (called by cron every 15 minutes)
|
||||
# watchdog_orchestrator.sh --dry-run — pass --dry-run to all sub-scripts
|
||||
# watchdog_orchestrator.sh --status — show script paths and current grace state
|
||||
# watchdog_orchestrator.sh --log — verbose output from all sub-scripts
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# watchdog_orchestrator.sh
|
||||
# Normal run (called by cron every 15 minutes).
|
||||
#
|
||||
# watchdog_orchestrator.sh --dry-run
|
||||
# Pass --dry-run to all sub-scripts.
|
||||
#
|
||||
# watchdog_orchestrator.sh --status
|
||||
# Show script paths and current grace state.
|
||||
#
|
||||
# watchdog_orchestrator.sh --log
|
||||
# Verbose output from all sub-scripts.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,57 +2,82 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Weekly Sync Maintenance ========================================
|
||||
# ==============================================================================================
|
||||
# Weekly maintenance window orchestrator — clean sync, container updates, weekly restarts.
|
||||
# Schedule: 30 2 * * 0 (Sunday 2:30am — before Sunday 7am coffee report)
|
||||
#
|
||||
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
|
||||
# 1. Stop local containers — Emby + auth stack stopped locally
|
||||
# 2. Stop remote containers — Emby + auth stack stopped remotely via SSH
|
||||
# 3. Pull updates locally — if WEEKLY_SYNC_UPDATES=true (zero extra downtime)
|
||||
# 4. Pull updates remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true
|
||||
# 5. rsync WEEKLY_SYNC_SHARES — full clean mirror, containers stopped both sides
|
||||
# 6. Start remote containers — correct order, delayed start respected
|
||||
# 7. Start local containers — rebuild if new image pulled, docker start otherwise
|
||||
# 8. WEEKLY_MAINTENANCE_SCRIPTS — weekly restarts etc. (docker_weekly_restart.sh)
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Weekly maintenance window orchestrator — clean Emby sync, container updates,
|
||||
# and weekly restarts. Schedule: 30 2 * * 0 (Sunday 2:30am)
|
||||
#
|
||||
# ── WHY WEEKLY NOT NIGHTLY FOR EMBY ──────────────────────────────────────────────────────────
|
||||
# Emby builds a warm image cache on HOST2 throughout the week.
|
||||
# Syncing nightly resets cache — cold loads every morning for users.
|
||||
# Weekly sync: cache stays warm 6 days, resets Sunday night while users sleep.
|
||||
# play_state_sync covers watch/resume state every 30 min between weekly syncs.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONTAINER UPDATES ─────────────────────────────────────────────────────────────────────────
|
||||
# Containers already stopped for sync — updates pull at zero extra downtime.
|
||||
# Both servers start on identical image versions after the window completes.
|
||||
# Toggle: WEEKLY_SYNC_UPDATES / WEEKLY_SYNC_UPDATES_REMOTE in master.conf
|
||||
# 1. Stop local containers — Emby + auth stack stopped locally
|
||||
# 2. Stop remote containers — Emby + auth stack stopped remotely via SSH
|
||||
# 3. Pull updates locally — if WEEKLY_SYNC_UPDATES=true (zero extra downtime)
|
||||
# 4. Pull updates remotely — if WEEKLY_SYNC_UPDATES_REMOTE=true
|
||||
# 5. rsync WEEKLY_SYNC_SHARES — full clean mirror, containers stopped both sides
|
||||
# 6. Start remote containers — correct order, delayed start respected
|
||||
# 7. Start local containers — rebuild if new image pulled, docker start otherwise
|
||||
# 8. WEEKLY_MAINTENANCE_SCRIPTS — weekly restarts etc. (docker_weekly_restart.sh)
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID — used in banner, summary, and notifications.
|
||||
# WEEKLY_SYNC_SHARES and WEEKLY_MAINTENANCE_SCRIPTS configured in master.conf.
|
||||
# Same script runs correctly on both servers.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — stop/start containers, rsync require root
|
||||
# acquire_lock — prevents concurrent weekly windows
|
||||
# check_connectivity — verifies remote before any remote operations
|
||||
# check_remote_rootfs — aborts if remote rootfs nearly full
|
||||
# DOCKER_TIMEOUT — all docker calls protected
|
||||
# SSH_TIMEOUT — all SSH calls protected
|
||||
# Weekly Cadence Preserves Cache
|
||||
# Emby builds a warm image cache on HOST2 throughout the week. Syncing nightly
|
||||
# resets that cache — cold loads every morning for users. Weekly sync keeps
|
||||
# the cache warm for 6 days, resets Sunday night while users sleep.
|
||||
# play_state_sync covers watch/resume state every 30 min between weekly syncs.
|
||||
#
|
||||
# Zero-Downtime Updates
|
||||
# Containers are already stopped for the sync window — image pulls happen at
|
||||
# zero extra downtime. Both servers start on identical image versions after
|
||||
# the window completes.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root check — stop/start containers and rsync require root
|
||||
# acquire_lock — prevents concurrent weekly windows
|
||||
# detect_hosts() — MY_ID in banner, summary, and notifications
|
||||
# check_connectivity — verifies remote before any remote operations
|
||||
# check_remote_rootfs — aborts rsync if remote rootfs nearly full
|
||||
# DOCKER_TIMEOUT — all docker calls protected
|
||||
# SSH_TIMEOUT — all SSH calls protected
|
||||
# platform_require_cmd — notify validated before use
|
||||
# Silent on success — runs weekly, only failures warrant notification
|
||||
# Silent on success — runs weekly; only failures warrant notification
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# WEEKLY_SYNC_SHARES — shares synced during window
|
||||
# WEEKLY_MAINTENANCE_SCRIPTS — scripts run after sync
|
||||
# WEEKLY_SYNC_UPDATES — toggle local container updates
|
||||
# WEEKLY_SYNC_UPDATES_REMOTE — toggle remote container updates
|
||||
# WEEKLY_RSYNC_ENABLED — enable/disable rsync section
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WEEKLY_SYNC_SHARES — shares synced during window
|
||||
# WEEKLY_MAINTENANCE_SCRIPTS — scripts run after sync
|
||||
# WEEKLY_SYNC_UPDATES — toggle local container updates
|
||||
# WEEKLY_SYNC_UPDATES_REMOTE — toggle remote container updates
|
||||
# WEEKLY_RSYNC_ENABLED — enable/disable rsync section
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# weekly_sync_maintenance.sh
|
||||
# Normal run.
|
||||
#
|
||||
# weekly_sync_maintenance.sh --dry-run
|
||||
# Preview without stopping containers or syncing.
|
||||
#
|
||||
# weekly_sync_maintenance.sh --log
|
||||
# Verbose per-share/per-job output.
|
||||
#
|
||||
# weekly_sync_maintenance.sh --status
|
||||
# Show configuration and exit.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# weekly_sync_maintenance.sh — normal run
|
||||
# weekly_sync_maintenance.sh --dry-run — preview without stopping containers or syncing
|
||||
# weekly_sync_maintenance.sh --log — verbose per-share/per-job output
|
||||
# weekly_sync_maintenance.sh --status — show configuration and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user