did way to much,,,,,, mostly added monitors, but almost every file was edited in some way
This commit is contained in:
+409
-218
@@ -5,6 +5,13 @@
|
|||||||
# All user-facing variables for the unRAID script ecosystem.
|
# All user-facing variables for the unRAID script ecosystem.
|
||||||
# Scripts source this file — edit here, changes apply everywhere on next git pull.
|
# Scripts source this file — edit here, changes apply everywhere on next git pull.
|
||||||
#
|
#
|
||||||
|
# ── HOW THIS FILE WORKS ───────────────────────────────────────────────────────────────────────
|
||||||
|
# Every script sources Master.conf and common.sh at startup.
|
||||||
|
# Change a value here and it affects all scripts that use it — no hunting through files.
|
||||||
|
# To disable something: comment it out with # rather than deleting it.
|
||||||
|
# To add a new rsync profile: add a key to each PROFILE_* array.
|
||||||
|
# To add a new media maintenance job: add a line to MEDIA_MAINTENANCE_JOBS.
|
||||||
|
#
|
||||||
# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
|
# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
|
||||||
#
|
#
|
||||||
# Section Description
|
# Section Description
|
||||||
@@ -36,47 +43,70 @@
|
|||||||
# PHP-FPM PHP-FPM max children config
|
# PHP-FPM PHP-FPM max children config
|
||||||
# CLEAR LOGS System log file paths
|
# CLEAR LOGS System log file paths
|
||||||
# WEBGUI WATCHDOG WebGUI nginx + emhttp monitoring and restart
|
# WEBGUI WATCHDOG WebGUI nginx + emhttp monitoring and restart
|
||||||
# ZFS MEMORY SNAPSHOT Weekly ZFS health and memory diagnostic report
|
|
||||||
#
|
#
|
||||||
# ── MEDIA ──────────────────────────────────────────────────────────────────────────────────
|
# ── MEDIA ──────────────────────────────────────────────────────────────────────────────────
|
||||||
# MEDIA PERMISSIONS Share list, mode and owner for permissions script
|
# MEDIA PERMISSIONS Share list, mode and owner for permissions script
|
||||||
# MEDIA CLEANER Anime and media folder lists and file patterns
|
# MEDIA CLEANER Anime and media folder lists and file patterns
|
||||||
|
# MEDIA MANAGEMENT Orchestrator job list for media_management.sh
|
||||||
|
# ARR CLEANUP Lidarr, Sonarr, Radarr orphan file cleanup
|
||||||
#
|
#
|
||||||
# ── TRANSCODES ─────────────────────────────────────────────────────────────────────────────
|
# ── TRANSCODES ─────────────────────────────────────────────────────────────────────────────
|
||||||
# TRANSCODE MANAGER Ramdisk and SSD fallback transcode management
|
# TRANSCODE MANAGER Ramdisk and SSD fallback transcode management
|
||||||
#
|
#
|
||||||
|
# ── MONITOR ────────────────────────────────────────────────────────────────────────────────
|
||||||
|
# CERTIFICATE MONITOR SSL certificate expiry monitoring
|
||||||
|
# BACKUP VERIFY Random sample checksum verification against remote
|
||||||
|
# SMART HEALTH Drive SMART attribute monitoring
|
||||||
|
# BANDWIDTH MONITOR Daily rsync transfer logging and weekly summary
|
||||||
|
# HEALTH DIGEST Aggregated system health digest — always/smart/weekly
|
||||||
|
# EMBY SESSION REPORT Weekly Emby usage statistics via API
|
||||||
|
#
|
||||||
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
|
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
|
||||||
# SYSTEM WATCHDOG System health monitoring — last line of defense
|
# SYSTEM WATCHDOG System health monitoring — last line of defense
|
||||||
#
|
#
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
|
|
||||||
# ━━━ Host Configuration ━━━
|
# ━━━ Host Configuration ━━━
|
||||||
# Hostnames must match Tailscale machine names exactly — case sensitive
|
# Hostnames must match Tailscale machine names exactly — case sensitive.
|
||||||
|
# Used by detect_hosts() in common.sh to determine which server is local and which is remote.
|
||||||
|
# Both servers run identical scripts — host detection makes them bidirectional.
|
||||||
HOST1="unRAID-Gmer4Lfe"
|
HOST1="unRAID-Gmer4Lfe"
|
||||||
HOST2="unRAID-Jayred365"
|
HOST2="unRAID-Jayred365"
|
||||||
# SSH keys for server-to-server rsync and failover operations
|
|
||||||
# Each server authenticates with its own key — both must be authorised on the remote
|
# SSH keys for server-to-server rsync and failover container operations.
|
||||||
|
# HOST1_SSH_KEY is used when HOST1 SSHes to HOST2 and vice versa.
|
||||||
|
# Both keys must be in /root/.ssh/ and authorised in the remote server's authorized_keys.
|
||||||
HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key"
|
HOST1_SSH_KEY="/root/.ssh/Gmer4Lfe-rsync-key"
|
||||||
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"
|
HOST2_SSH_KEY="/root/.ssh/Jayred365-rsync-key"
|
||||||
|
|
||||||
# ━━━ Logging ━━━
|
# ━━━ Logging ━━━
|
||||||
# true = verbose [LOG] lines in all script output / false = user-facing output only
|
# Controls verbose [LOG] output across all scripts.
|
||||||
|
# true = show detailed [LOG] lines — useful for debugging or first-time setup
|
||||||
|
# false = show only user-facing output — cleaner for scheduled runs
|
||||||
ENABLE_LOGGING=true
|
ENABLE_LOGGING=true
|
||||||
|
|
||||||
# ━━━ Notifications ━━━
|
# ━━━ Notifications ━━━
|
||||||
# unRAID native — fires through Settings → Notification Settings
|
# Two independent notification channels — either or both can be active simultaneously.
|
||||||
# Recommended: configure unRAID to send errors/warnings only so normal completions stay quiet
|
|
||||||
|
# unRAID native notification system — integrates with the bell icon in the WebGUI.
|
||||||
|
# Recommended: set Settings → Notification Settings to errors/warnings only so
|
||||||
|
# normal completions don't create noise. The ecosystem sends:
|
||||||
|
# normal — job completed successfully (informational)
|
||||||
|
# warning — something failed or needs attention
|
||||||
NOTIFY_UNRAID=true
|
NOTIFY_UNRAID=true
|
||||||
# Discord webhook — paste full webhook URL to enable, leave blank to disable
|
|
||||||
|
# Discord webhook URL — paste the full webhook URL from your Discord server settings.
|
||||||
|
# Leave blank to disable Discord notifications entirely.
|
||||||
DISCORD_WEBHOOK=""
|
DISCORD_WEBHOOK=""
|
||||||
|
|
||||||
# ━━━ Git / Repo ━━━
|
# ━━━ Git / Repo ━━━
|
||||||
# Gitea self-hosted repository for the script ecosystem
|
# Gitea self-hosted repository settings used by git_pull_execute.sh.
|
||||||
# git_pull_execute.sh uses these to clone or pull the latest version on both servers
|
# Running git_pull_execute.sh on either server pulls the latest scripts and sets
|
||||||
|
# executable permissions automatically — keeps both servers in sync with one command.
|
||||||
REPO_SSH="git@192.168.50.2:FailedProxy/Unraid_Scripts.git"
|
REPO_SSH="git@192.168.50.2:FailedProxy/Unraid_Scripts.git"
|
||||||
TARGET_DIR="/mnt/user/appdata/unraid_scripts"
|
TARGET_DIR="/mnt/user/appdata/unraid_scripts"
|
||||||
GITEA_SSH_KEY="/root/.ssh/unraid_gitea"
|
GITEA_SSH_KEY="/root/.ssh/unraid_gitea" # SSH key for authenticating to Gitea
|
||||||
SSH_PORT=221
|
SSH_PORT=221 # Gitea SSH port — default Gitea uses 22
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── RSYNC ─────────────────────────────────────────────────────────────────────────────────────
|
# ── RSYNC ─────────────────────────────────────────────────────────────────────────────────────
|
||||||
@@ -86,39 +116,35 @@
|
|||||||
# Global fallback values used when no profile match is found for a directory.
|
# Global fallback values used when no profile match is found for a directory.
|
||||||
# Shares in DAILY_SYNC_SHARES always use these globals — no profile is defined for them.
|
# Shares in DAILY_SYNC_SHARES always use these globals — no profile is defined for them.
|
||||||
# Appdata shares (Arrs_Stack, Critical-Data etc.) match profiles by directory basename.
|
# Appdata shares (Arrs_Stack, Critical-Data etc.) match profiles by directory basename.
|
||||||
|
# If a profile key exists in a PROFILE_* array that value overrides the global.
|
||||||
|
# If a profile key is missing the global below is used as the fallback.
|
||||||
|
|
||||||
# Network transfer speed cap in KB/s — 12500 ≈ 100Mbit
|
BW_LIMIT=12500 # network transfer speed cap in KB/s — 12500 ≈ 100Mbit
|
||||||
BW_LIMIT=12500
|
RETRY_COUNT=3 # number of retry attempts if rsync fails before giving up
|
||||||
# Number of retry attempts if rsync fails before giving up
|
SLEEP=300 # seconds to wait between retry attempts
|
||||||
RETRY_COUNT=3
|
CRITICAL_CONTAINER_NAMES=() # containers to stop on REMOTE before rsync — profiles override
|
||||||
# Seconds to wait between retry attempts
|
DELAYED_CONTAINERS=() # containers needing delay before starting — profiles override
|
||||||
SLEEP=300
|
CONTAINER_DELAY=5 # seconds to wait before starting delayed containers
|
||||||
# Containers to stop on remote before rsync and restart after — empty by default
|
EXCLUDE_DIRS=() # directories to exclude from transfer — profiles override
|
||||||
# Profiles below override this per appdata share
|
|
||||||
CRITICAL_CONTAINER_NAMES=()
|
# Default rsync options used when no profile match is found.
|
||||||
# Containers that need a delay before starting — e.g. Authelia needs its DB ready first
|
# --delete removes files on remote that no longer exist on source (mirror behaviour)
|
||||||
DELAYED_CONTAINERS=()
|
# --inplace writes directly to destination file — better for large files, avoids temp copies
|
||||||
# Seconds to wait before starting delayed containers
|
# --no-whole-file forces delta transfer even on fast local-like connections
|
||||||
CONTAINER_DELAY=5
|
|
||||||
# Directories to exclude from transfer — empty by default, profiles override per share
|
|
||||||
EXCLUDE_DIRS=()
|
|
||||||
# Default rsync options used when no profile match is found
|
|
||||||
# --delete removes files on remote that no longer exist on source
|
|
||||||
# --inplace writes directly to destination file rather than temp file — better for large files
|
|
||||||
# --no-whole-file forces delta transfer even on local-like connections
|
|
||||||
DEFAULT_RSYNC_OPTS=(-av --info=progress2 --human-readable --bwlimit="$BW_LIMIT" --delete --inplace --no-whole-file)
|
DEFAULT_RSYNC_OPTS=(-av --info=progress2 --human-readable --bwlimit="$BW_LIMIT" --delete --inplace --no-whole-file)
|
||||||
|
|
||||||
# ━━━ Remote Health Checks ━━━
|
# ━━━ Remote Health Checks ━━━
|
||||||
# Abort rsync if remote rootfs (/) usage is at or above this percentage.
|
# Pre-flight check run before every rsync — aborts if remote rootfs (/) usage is at or
|
||||||
# When the remote array is down or drives are missing, rsync writes land on rootfs instead
|
# above this percentage. When the remote array is down or drives are missing, rsync
|
||||||
# of /mnt/user — this fills the filesystem rapidly and can crash the remote server.
|
# writes land on rootfs instead of /mnt/user — this fills the filesystem rapidly and
|
||||||
# 75% gives enough headroom to detect the problem before it becomes critical.
|
# can crash the remote server. 75% gives headroom to detect the problem early.
|
||||||
ROOTFS_WARN=75
|
ROOTFS_WARN=75
|
||||||
|
|
||||||
# ━━━ Daily Sync Shares ━━━
|
# ━━━ Daily Sync Shares ━━━
|
||||||
# Media shares synced once daily by Orchestrators/daily_sync.sh
|
# Media shares synced once daily by Orchestrators/daily_sync.sh.
|
||||||
# These shares have no profile — all use DEFAULT_RSYNC_OPTS above
|
# These shares have no profile entry — all use DEFAULT_RSYNC_OPTS above.
|
||||||
# Add or remove paths to control what gets synced each night
|
# Add or remove paths here to control what syncs each night.
|
||||||
|
# For shares needing custom bandwidth or container stops — create a profile below instead.
|
||||||
DAILY_SYNC_SHARES=(
|
DAILY_SYNC_SHARES=(
|
||||||
/mnt/user/Anime_Movies-Old
|
/mnt/user/Anime_Movies-Old
|
||||||
/mnt/user/Anime_Shows-Old
|
/mnt/user/Anime_Shows-Old
|
||||||
@@ -136,36 +162,38 @@ DAILY_SYNC_SHARES=(
|
|||||||
|
|
||||||
# ━━━ Rsync Profile System ━━━
|
# ━━━ Rsync Profile System ━━━
|
||||||
# Profiles allow per-share rsync behaviour without touching script logic.
|
# Profiles allow per-share rsync behaviour without touching script logic.
|
||||||
# Each profile is matched automatically by the basename of the directory passed to rsync.sh
|
# The profile key is matched automatically by the basename of the directory
|
||||||
# (lowercased). Example: /mnt/user/appdata-Failover/Arrs_Stack → profile key = arrs_stack
|
# passed to rsync.sh (lowercased).
|
||||||
#
|
#
|
||||||
# How fallthrough works:
|
# Example:
|
||||||
# If a key exists in a profile array → that value is used for this run
|
# rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
|
||||||
# If a key is missing → the global default above is used instead
|
# basename = Arrs_Stack → lowercased = arrs_stack → matches [arrs_stack] profile
|
||||||
# Shares in DAILY_SYNC_SHARES → always use globals, no profile defined
|
|
||||||
#
|
#
|
||||||
# To add a new profile:
|
# To add a new profile:
|
||||||
# 1. Add a key to each array below with your chosen profile name
|
# 1. Add a key to each PROFILE_* array below using your chosen name
|
||||||
# 2. Call rsync.sh with a directory whose basename matches that key
|
# 2. Call rsync.sh with a directory whose basename matches that key
|
||||||
# 3. Any array you omit falls back to its global default automatically
|
# 3. Any array you omit falls back to its global default automatically
|
||||||
#
|
#
|
||||||
# IMPORTANT: PROFILE_RSYNC_OPTS does NOT inherit from DEFAULT_RSYNC_OPTS —
|
# IMPORTANT: PROFILE_RSYNC_OPTS does NOT inherit from DEFAULT_RSYNC_OPTS.
|
||||||
# if you define a profile entry you must list all desired options explicitly
|
# If you define a profile entry you must list ALL desired options explicitly.
|
||||||
#
|
#
|
||||||
# Profile descriptions:
|
# Current profiles:
|
||||||
# arrs_stack — Sonarr, Radarr, Lidarr, Readarr, Prowlarr, Bazarr, Pinchflat appdata
|
# arrs_stack — Sonarr, Radarr, Lidarr, Readarr, Prowlarr, Bazarr, Pinchflat
|
||||||
# Lower bandwidth — runs alongside media syncs, containers stopped during sync
|
# Lower bandwidth — runs alongside media syncs
|
||||||
# critical-data — Auth stack: NPM, Authelia, Mariadb, Redis, LLDAP
|
# Containers stopped during sync for data consistency
|
||||||
# High bandwidth — small data, synced frequently, Authelia needs delayed start
|
# critical-data — Auth stack: NPM, Authelia, Mariadb-Authelia, Redis-Authelia, LLDAP
|
||||||
|
# High bandwidth — small data, synced frequently
|
||||||
|
# Authelia needs delayed start — database containers must be ready first
|
||||||
# gmer4lfe — Server-specific appdata: Organizr, UptimeKuma, VaultWarden
|
# gmer4lfe — Server-specific appdata: Organizr, UptimeKuma, VaultWarden
|
||||||
# Medium bandwidth — personal services, no container stop needed
|
# Medium bandwidth — personal services, no container stop needed
|
||||||
# important-data — NextCloud + Postgres database
|
# important-data — NextCloud + Postgres database
|
||||||
# High bandwidth — NextCloud needs graceful stop before sync
|
# High bandwidth — NextCloud needs graceful stop before sync
|
||||||
# emby — Emby media server appdata and metadata
|
# NextCloud needs delayed start — Postgres must be accepting connections
|
||||||
# Medium bandwidth — large appdata, no containers stopped (metadata only)
|
# emby — Emby media server appdata and metadata only
|
||||||
|
# Medium bandwidth — large appdata directory, no containers stopped
|
||||||
|
|
||||||
# SPACE-SEPARATED STRINGS — rsync options per profile
|
# Rsync options per profile — replaces DEFAULT_RSYNC_OPTS entirely for that profile run
|
||||||
# If defined for a profile, these replace DEFAULT_RSYNC_OPTS entirely for that run
|
# SPACE-SEPARATED STRINGS — converted to array at runtime by rsync.sh
|
||||||
declare -A PROFILE_RSYNC_OPTS=(
|
declare -A PROFILE_RSYNC_OPTS=(
|
||||||
[arrs_stack]="-av --info=progress2 --human-readable --bwlimit=$BW_LIMIT --delete --inplace"
|
[arrs_stack]="-av --info=progress2 --human-readable --bwlimit=$BW_LIMIT --delete --inplace"
|
||||||
[critical-data]="-av --human-readable --bwlimit=$BW_LIMIT --delete"
|
[critical-data]="-av --human-readable --bwlimit=$BW_LIMIT --delete"
|
||||||
@@ -174,16 +202,16 @@ declare -A PROFILE_RSYNC_OPTS=(
|
|||||||
[emby]="-av --human-readable --bwlimit=$BW_LIMIT"
|
[emby]="-av --human-readable --bwlimit=$BW_LIMIT"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Bandwidth limit in KB/s per profile — overrides global BW_LIMIT for this profile
|
# Bandwidth limit in KB/s per profile — overrides global BW_LIMIT for this profile only
|
||||||
declare -A PROFILE_BW_LIMIT=(
|
declare -A PROFILE_BW_LIMIT=(
|
||||||
[arrs_stack]=5000 # lower — runs alongside other jobs
|
[arrs_stack]=5000 # lower — runs alongside other jobs, avoids saturating link
|
||||||
[critical-data]=9500 # high — small data, sync fast
|
[critical-data]=9500 # high — small data, get it synced fast
|
||||||
[gmer4lfe]=8000 # medium
|
[gmer4lfe]=8000 # medium
|
||||||
[important-data]=9500 # high — database sync, prioritise speed
|
[important-data]=9500 # high — database sync needs to be fast and clean
|
||||||
[emby]=8000 # medium — large files, steady transfer
|
[emby]=8000 # medium — large files, steady sustained transfer
|
||||||
)
|
)
|
||||||
|
|
||||||
# Retry attempts per profile — overrides global RETRY_COUNT
|
# Retry attempts per profile before giving up — overrides global RETRY_COUNT
|
||||||
declare -A PROFILE_RETRY_COUNT=(
|
declare -A PROFILE_RETRY_COUNT=(
|
||||||
[arrs_stack]=3
|
[arrs_stack]=3
|
||||||
[critical-data]=3
|
[critical-data]=3
|
||||||
@@ -192,7 +220,7 @@ declare -A PROFILE_RETRY_COUNT=(
|
|||||||
[emby]=3
|
[emby]=3
|
||||||
)
|
)
|
||||||
|
|
||||||
# Sleep between retries in seconds — overrides global SLEEP
|
# Seconds between retry attempts — overrides global SLEEP
|
||||||
declare -A PROFILE_SLEEP=(
|
declare -A PROFILE_SLEEP=(
|
||||||
[arrs_stack]=300
|
[arrs_stack]=300
|
||||||
[critical-data]=300
|
[critical-data]=300
|
||||||
@@ -201,9 +229,10 @@ declare -A PROFILE_SLEEP=(
|
|||||||
[emby]=300
|
[emby]=300
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers stopped on REMOTE before rsync and restarted after — SPACE-SEPARATED STRINGS
|
# Containers stopped on REMOTE before rsync and restarted after completion.
|
||||||
# Only containers that need to be stopped for data consistency — running containers are fine
|
# Only include containers that need to be stopped for data consistency.
|
||||||
# for most media but databases and auth stacks need clean state during sync
|
# Databases and auth stacks need clean state — media servers generally do not.
|
||||||
|
# SPACE-SEPARATED STRINGS — converted to array at runtime
|
||||||
declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
|
declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
|
||||||
[arrs_stack]="Sonarr Lidarr Readarr Radarr Prowlarr Bazarr Pinchflat"
|
[arrs_stack]="Sonarr Lidarr Readarr Radarr Prowlarr Bazarr Pinchflat"
|
||||||
[critical-data]="Mariadb-Authelia Redis-Authelia Lldap-Gmer4Lfe NginxProxyManager Authelia"
|
[critical-data]="Mariadb-Authelia Redis-Authelia Lldap-Gmer4Lfe NginxProxyManager Authelia"
|
||||||
@@ -212,28 +241,30 @@ declare -A PROFILE_CRITICAL_CONTAINER_NAMES=(
|
|||||||
[emby]=""
|
[emby]=""
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers that need a delay before starting after rsync — SPACE-SEPARATED STRINGS
|
# Containers that need a delay before starting after rsync completes.
|
||||||
# Authelia needs its database containers (Mariadb, Redis) to be ready before it starts
|
# Used when a container depends on another that was also stopped — it needs its
|
||||||
# NextCloud needs Postgres to be accepting connections before it starts
|
# dependency to be ready before it can start successfully.
|
||||||
|
# SPACE-SEPARATED STRINGS — converted to array at runtime
|
||||||
declare -A PROFILE_DELAYED_CONTAINERS=(
|
declare -A PROFILE_DELAYED_CONTAINERS=(
|
||||||
[arrs_stack]=""
|
[arrs_stack]=""
|
||||||
[critical-data]="Authelia"
|
[critical-data]="Authelia" # Authelia needs Mariadb + Redis ready before starting
|
||||||
[gmer4lfe]=""
|
[gmer4lfe]=""
|
||||||
[important-data]="NextCloud"
|
[important-data]="NextCloud" # NextCloud needs Postgres accepting connections first
|
||||||
[emby]=""
|
[emby]=""
|
||||||
)
|
)
|
||||||
|
|
||||||
# Seconds to wait before starting delayed containers — per profile
|
# Seconds to wait before starting delayed containers — gives dependencies time to initialise
|
||||||
declare -A PROFILE_CONTAINER_DELAY=(
|
declare -A PROFILE_CONTAINER_DELAY=(
|
||||||
[arrs_stack]=5
|
[arrs_stack]=5
|
||||||
[critical-data]=10 # Authelia needs DB ready — 10s gives Mariadb/Redis time to start
|
[critical-data]=10 # 10s gives Mariadb and Redis time to accept connections
|
||||||
[gmer4lfe]=5
|
[gmer4lfe]=5
|
||||||
[important-data]=10 # NextCloud needs Postgres ready
|
[important-data]=10 # 10s gives Postgres time to accept connections
|
||||||
[emby]=5
|
[emby]=5
|
||||||
)
|
)
|
||||||
|
|
||||||
# Directories excluded from transfer per profile — SPACE-SEPARATED STRINGS
|
# Directories excluded from rsync transfer per profile.
|
||||||
# logs and *.tmp are excluded universally — they are ephemeral and regenerated on start
|
# logs and *.tmp are safe to exclude — they are ephemeral and regenerated on container start.
|
||||||
|
# SPACE-SEPARATED STRINGS — converted to array at runtime
|
||||||
declare -A PROFILE_EXCLUDE_DIRS=(
|
declare -A PROFILE_EXCLUDE_DIRS=(
|
||||||
[arrs_stack]="logs *.tmp"
|
[arrs_stack]="logs *.tmp"
|
||||||
[critical-data]="logs *.tmp"
|
[critical-data]="logs *.tmp"
|
||||||
@@ -242,75 +273,78 @@ declare -A PROFILE_EXCLUDE_DIRS=(
|
|||||||
[emby]="logs *.tmp"
|
[emby]="logs *.tmp"
|
||||||
)
|
)
|
||||||
|
|
||||||
# ━━━ Profile Disk Check Toggle ━━━
|
# Per-disk check toggle — controls whether rsync.sh runs check_remote_disks() for this profile.
|
||||||
# true = skip per-disk check for this profile — use when share lives on a ZFS pool
|
# true = skip per-disk check — use when remote share lives on a ZFS pool
|
||||||
# false = run per-disk check — use for traditional unRAID array with individual disks
|
# ZFS pools don't have /mnt/disk* structure so the check always fails incorrectly
|
||||||
# Per-disk check looks for /mnt/disk*/sharename — ZFS pools don't have this structure
|
# false = run per-disk check — use for traditional unRAID array with individual disk mounts
|
||||||
# The share existence and content checks still run regardless of this setting
|
# Verifies all disks backing the share are online before syncing
|
||||||
|
# Note: rootfs and share existence checks always run regardless of this setting
|
||||||
declare -A PROFILE_SKIP_DISK_CHECK=(
|
declare -A PROFILE_SKIP_DISK_CHECK=(
|
||||||
[arrs_stack]=true
|
[arrs_stack]=true # remote uses ZFS pool — no individual disk mounts
|
||||||
[critical-data]=true
|
[critical-data]=true
|
||||||
[gmer4lfe]=true
|
[gmer4lfe]=true
|
||||||
[important-data]=true
|
[important-data]=true
|
||||||
[emby]=true
|
[emby]=true
|
||||||
)
|
)
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── FAILOVER ──────────────────────────────────────────────────────────────────────────────────
|
# ── FAILOVER ──────────────────────────────────────────────────────────────────────────────────
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# Mutual container failover between two unRAID servers 50 miles apart.
|
# Mutual container failover between two unRAID servers 50 miles apart.
|
||||||
# Each server runs failover.sh independently — no coordination between servers.
|
# Each server runs Failover/failover.sh independently — no coordination between servers.
|
||||||
# All decisions are based solely on two ping checks: remote reachable + internet reachable.
|
# All decisions are based solely on two ping checks: remote reachable + internet reachable.
|
||||||
#
|
#
|
||||||
|
# How it works:
|
||||||
|
# Every FAILOVER_CHECK_INTERVAL seconds each server pings the remote and pings the internet.
|
||||||
|
# Based on those two results it determines its state and takes the appropriate action.
|
||||||
|
# No SSH, no signaling — each server acts autonomously based only on what it can see.
|
||||||
|
#
|
||||||
# States:
|
# States:
|
||||||
# NORMAL — remote up, internet up — own containers only, silent operation
|
# NORMAL — remote up, internet up — own containers only, silent operation
|
||||||
# FAILOVER — remote down, internet up — start remote's containers locally (additive)
|
# FAILOVER — remote down, internet up — start remote's containers locally (additive)
|
||||||
# NO_INTERNET — internet down — stop public-facing containers, wait
|
# Own normal containers keep running — failover containers added on top
|
||||||
# DARK — remote down + internet down — same as NO_INTERNET
|
# NO_INTERNET — internet down — stop public-facing containers, wait for recovery
|
||||||
|
# DARK — remote down + internet down — same actions as NO_INTERNET
|
||||||
#
|
#
|
||||||
# Handback sequence when remote returns after FAILOVER:
|
# Handback sequence when remote returns after FAILOVER:
|
||||||
# Strike confirmation (FAILOVER_HANDBACK_STRIKES consecutive remote-up checks)
|
# 1. Strike confirmation — FAILOVER_HANDBACK_STRIKES consecutive remote-up checks
|
||||||
# → pre-flight checks (remote array, docker daemon, rootfs)
|
# Prevents handing back during a brief network blip
|
||||||
# → rsync data back via rsync.sh (uses profile system)
|
# 2. Pre-flight checks — remote array started, Docker daemon healthy, rootfs not full
|
||||||
# → start failover containers on remote via SSH
|
# 3. Rsync data back via rsync.sh — uses existing profile system for options
|
||||||
# → stop failover containers locally
|
# 4. Start failover containers on remote via SSH
|
||||||
# → return to NORMAL
|
# 5. Stop failover containers locally — only after remote confirmed started
|
||||||
|
# 6. Return to NORMAL state
|
||||||
#
|
#
|
||||||
# Comment out any container or rsync job to disable without removing the entry.
|
|
||||||
# Script runs on BOTH servers — detect_hosts() selects the correct arrays automatically.
|
# Script runs on BOTH servers — detect_hosts() selects the correct arrays automatically.
|
||||||
|
# Comment out any container or rsync job to disable without removing the entry.
|
||||||
|
|
||||||
# External IP to ping for internet connectivity check — Google DNS, reliable and fast
|
EXTERNAL_IP="8.8.8.8" # external IP to ping for internet connectivity check
|
||||||
EXTERNAL_IP="8.8.8.8"
|
FAILOVER_CHECK_INTERVAL=120 # seconds between state checks — 120s = 2 minute polling
|
||||||
# Seconds between state checks — 120s = 2 minute polling interval
|
FAILOVER_HANDBACK_STRIKES=2 # consecutive remote-up confirmations required before handback
|
||||||
# With 1 minute DNS TTL this means failover is visible to users within ~3 minutes
|
# 2 strikes at 120s interval = 4 minutes confirmation window
|
||||||
FAILOVER_CHECK_INTERVAL=120
|
|
||||||
# Consecutive remote-up confirmations required before initiating handback
|
|
||||||
# Prevents handing back during a brief network blip — 2 strikes = 4 minutes confirmation
|
|
||||||
FAILOVER_HANDBACK_STRIKES=2
|
|
||||||
# State file path — on /boot/ so it survives reboots
|
|
||||||
# Script re-evaluates from scratch on restart using live pings — state file is reference only
|
|
||||||
FAILOVER_STATE_FILE="/boot/config/failover_state.db"
|
FAILOVER_STATE_FILE="/boot/config/failover_state.db"
|
||||||
|
# persists on /boot/ so it survives reboots
|
||||||
|
# script re-evaluates from live pings on restart
|
||||||
|
|
||||||
# ━━━ HOST1 Failover Config (unRAID-Gmer4Lfe — Primary) ━━━
|
# ━━━ HOST1 Failover Config (unRAID-Gmer4Lfe — Primary) ━━━
|
||||||
|
|
||||||
# Containers HOST1 starts locally when HOST2 goes down
|
# Containers HOST1 starts locally when HOST2 goes down.
|
||||||
# These run on top of HOST1's normal containers — additive, not replacement
|
# These run ON TOP OF HOST1's normal containers — additive, not a replacement.
|
||||||
# HOST2's DDNS containers must start here so DNS stays pointing at HOST1 during HOST2 outage
|
|
||||||
FAILOVER_HOST1_STARTS_FOR_HOST2=(
|
FAILOVER_HOST1_STARTS_FOR_HOST2=(
|
||||||
"Gmer4Lfe.com"
|
"Gmer4Lfe.com"
|
||||||
"Gmer4Lfe.us"
|
"Gmer4Lfe.us"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers HOST1 stops when it loses internet
|
# Containers HOST1 stops when it loses internet connectivity.
|
||||||
# No point serving DDNS or media if HOST1 itself has no internet — stops unnecessary churn
|
# No point serving DDNS or public services if HOST1 itself has no internet.
|
||||||
# HOST2 will independently detect HOST1 is gone and start its own failover
|
|
||||||
FAILOVER_HOST1_STOP_ON_NO_NET=(
|
FAILOVER_HOST1_STOP_ON_NO_NET=(
|
||||||
"Gmer4Lfe.com"
|
"Gmer4Lfe.com"
|
||||||
"Gmer4Lfe.us"
|
"Gmer4Lfe.us"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Rsync jobs HOST1 runs before handing containers back to HOST2
|
# Rsync jobs HOST1 runs before handing containers back to HOST2 after recovery.
|
||||||
# Uses rsync.sh with existing profile system — basename matched to profile keys
|
# Uses rsync.sh with the existing profile system — basename matched to profile keys.
|
||||||
# Comment out jobs that don't need syncing back or aren't ready yet
|
# Comment out jobs that are not yet ready or not needed for handback.
|
||||||
FAILOVER_HOST1_RSYNC_JOBS=(
|
FAILOVER_HOST1_RSYNC_JOBS=(
|
||||||
# "/mnt/user/appdata-Failover/Jayred365"
|
# "/mnt/user/appdata-Failover/Jayred365"
|
||||||
# "/mnt/user/Media_Server/Emby-Jayred"
|
# "/mnt/user/Media_Server/Emby-Jayred"
|
||||||
@@ -318,24 +352,22 @@ FAILOVER_HOST1_RSYNC_JOBS=(
|
|||||||
|
|
||||||
# ━━━ HOST2 Failover Config (unRAID-Jayred365 — Secondary) ━━━
|
# ━━━ HOST2 Failover Config (unRAID-Jayred365 — Secondary) ━━━
|
||||||
|
|
||||||
# Containers HOST2 starts locally when HOST1 goes down
|
# Containers HOST2 starts locally when HOST1 goes down.
|
||||||
# Emby starts on HOST2 so media keeps working during HOST1 outage
|
# Emby starts on HOST2 so media keeps working during HOST1 outage.
|
||||||
# HOST1's DDNS containers start here so DNS updates to point at HOST2
|
# HOST1's DDNS containers start here so DNS updates to point at HOST2's IP.
|
||||||
FAILOVER_HOST2_STARTS_FOR_HOST1=(
|
FAILOVER_HOST2_STARTS_FOR_HOST1=(
|
||||||
"Emby"
|
"Emby"
|
||||||
"Gmer4Lfe.com"
|
"Gmer4Lfe.com"
|
||||||
"Gmer4Lfe.us"
|
"Gmer4Lfe.us"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers HOST2 stops when it loses internet
|
# Containers HOST2 stops when it loses internet connectivity.
|
||||||
# HOST2's DDNS containers serve no purpose without internet connectivity
|
|
||||||
FAILOVER_HOST2_STOP_ON_NO_NET=(
|
FAILOVER_HOST2_STOP_ON_NO_NET=(
|
||||||
"Gmer4Lfe.com"
|
"Gmer4Lfe.com"
|
||||||
"Gmer4Lfe.us"
|
"Gmer4Lfe.us"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Rsync jobs HOST2 runs before handing containers back to HOST1
|
# Rsync jobs HOST2 runs before handing containers back to HOST1 after recovery.
|
||||||
# Comment out jobs that don't need syncing back or aren't ready yet
|
|
||||||
FAILOVER_HOST2_RSYNC_JOBS=(
|
FAILOVER_HOST2_RSYNC_JOBS=(
|
||||||
# "/mnt/user/appdata-Failover/Gmer4Lfe"
|
# "/mnt/user/appdata-Failover/Gmer4Lfe"
|
||||||
)
|
)
|
||||||
@@ -345,8 +377,9 @@ FAILOVER_HOST2_RSYNC_JOBS=(
|
|||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
|
|
||||||
# ━━━ Docker Daily Restart ━━━
|
# ━━━ Docker Daily Restart ━━━
|
||||||
# Containers restarted every day — keeps services fresh, clears memory leaks
|
# Containers restarted every day by Docker_Essentials/docker_daily_restart.sh.
|
||||||
# Case-sensitive — must match exact Docker container names
|
# Keeps services fresh and clears memory leaks that accumulate over time.
|
||||||
|
# Case-sensitive — must match exact Docker container names shown in the unRAID Docker tab.
|
||||||
DAILY_RESTART_CONTAINERS=(
|
DAILY_RESTART_CONTAINERS=(
|
||||||
"NginxProxyManager"
|
"NginxProxyManager"
|
||||||
"Authelia"
|
"Authelia"
|
||||||
@@ -357,7 +390,8 @@ DAILY_RESTART_CONTAINERS=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# ━━━ Docker Weekly Restart ━━━
|
# ━━━ Docker Weekly Restart ━━━
|
||||||
# Containers restarted once per week — less critical services that benefit from periodic restart
|
# Containers restarted once per week by Docker_Essentials/docker_weekly_restart.sh.
|
||||||
|
# For less critical services that benefit from periodic restart but don't need daily cycling.
|
||||||
WEEKLY_RESTART_CONTAINERS=(
|
WEEKLY_RESTART_CONTAINERS=(
|
||||||
"NextCloud"
|
"NextCloud"
|
||||||
"Organizrv2-Gmer4Lfe"
|
"Organizrv2-Gmer4Lfe"
|
||||||
@@ -366,30 +400,33 @@ WEEKLY_RESTART_CONTAINERS=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# ━━━ Docker Watchdog ━━━
|
# ━━━ Docker Watchdog ━━━
|
||||||
# First line of defense — monitors containers for memory, CPU and HTTP responsiveness.
|
# First line of defense for container health — runs every 15 minutes via cron.
|
||||||
# Restarts containers that exceed thresholds using a strike system to avoid false positives.
|
# Monitors memory usage, CPU usage and HTTP responsiveness per container.
|
||||||
# Works alongside system_watchdog.sh — containers first, system reboot second.
|
# Uses a strike system to avoid restarting on brief spikes — sustained issues trigger restart.
|
||||||
|
# Works alongside system_watchdog.sh — containers first, system reboot is the last resort.
|
||||||
|
|
||||||
# Containers to monitor with their memory hard limits in MB
|
# Containers to monitor with their memory hard limits in MB.
|
||||||
# Strike system used for CPU and responsiveness — immediate restart for memory hard limit
|
# Memory hard limit exceeded → immediate restart (no strike system for memory).
|
||||||
|
# CPU and HTTP use strike system — see CPU_FAIL_LIMIT and RESP_FAIL_LIMIT below.
|
||||||
# 20GB=20480 16GB=16384 14GB=14336 12GB=12288 10GB=10240
|
# 20GB=20480 16GB=16384 14GB=14336 12GB=12288 10GB=10240
|
||||||
# 8GB=8192 6GB=6144 4GB=4096 2GB=2048 1GB=1024
|
# 8GB=8192 6GB=6144 4GB=4096 2GB=2048 1GB=1024
|
||||||
declare -A WATCHDOG_CONTAINERS=(
|
declare -A WATCHDOG_CONTAINERS=(
|
||||||
["Emby"]=16384 # 16GB — large media server, transcoding can spike
|
["Emby"]=16384 # 16GB — media server, transcoding can spike high
|
||||||
["LidaTube"]=6144 # 6GB — YouTube downloader
|
["LidaTube"]=6144 # 6GB — YouTube downloader
|
||||||
["Tdarr"]=6144 # 6GB — transcoding node
|
["Tdarr"]=6144 # 6GB — transcoding node
|
||||||
["Code-Server"]=1024 # 1GB — VS Code server
|
["Code-Server"]=1024 # 1GB — VS Code server
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers to check HTTP responsiveness — curl check per interval
|
# Containers to check HTTP responsiveness via curl — omit a container to skip its HTTP check.
|
||||||
# Omit a container entirely to skip its HTTP check
|
# curl checks the URL and considers the container unresponsive if it times out or errors.
|
||||||
declare -A WATCHDOG_CONTAINER_URLS=(
|
declare -A WATCHDOG_CONTAINER_URLS=(
|
||||||
["Emby"]="http://localhost:8096"
|
["Emby"]="http://localhost:8096"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Containers that should always be running — monitored for unexpected stops
|
# Containers that should always be running — monitored for unexpected stops.
|
||||||
# Strike system used — persistent skip list on /boot/ prevents reboot loops
|
# Strike system used — tries restart on each strike up to SYS_WATCHDOG_STRIKE_LIMIT.
|
||||||
# Auto-clears from skip list when container recovers after reboot or manual fix
|
# If restart fails after strike limit → added to persistent skip list on /boot/
|
||||||
|
# Skip list auto-clears when container recovers after reboot or manual fix.
|
||||||
WATCHDOG_REQUIRED_CONTAINERS=(
|
WATCHDOG_REQUIRED_CONTAINERS=(
|
||||||
"NginxProxyManager"
|
"NginxProxyManager"
|
||||||
"Lldap-Gmer4Lfe"
|
"Lldap-Gmer4Lfe"
|
||||||
@@ -400,33 +437,36 @@ WATCHDOG_REQUIRED_CONTAINERS=(
|
|||||||
"Redis-Authelia-Secondary"
|
"Redis-Authelia-Secondary"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Strike state file — /tmp resets on reboot which is correct for strike tracking
|
# Strike state file — /tmp resets on reboot which is correct behaviour for strike tracking
|
||||||
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
|
WATCHDOG_STATE_FILE="/tmp/container_watchdog_state.db"
|
||||||
|
|
||||||
# CPU thresholds — normalised against total core count at runtime
|
# CPU thresholds — normalised against total core count automatically at runtime.
|
||||||
|
# A container using 85% of one core on a 16-core system = ~5.3% normalised.
|
||||||
SOFT_CPU_THRESHOLD=80 # warn at this % of total system CPU
|
SOFT_CPU_THRESHOLD=80 # warn at this % of total system CPU
|
||||||
HARD_CPU_THRESHOLD=85 # strike at this % of total system CPU
|
HARD_CPU_THRESHOLD=85 # strike at this % of total system CPU
|
||||||
CPU_FAIL_LIMIT=2 # consecutive strikes before container restart
|
CPU_FAIL_LIMIT=2 # consecutive strikes before container restart
|
||||||
|
|
||||||
# Memory — warn at this % of per-container hard limit defined in WATCHDOG_CONTAINERS
|
# Memory soft threshold — warn when container reaches this % of its hard limit.
|
||||||
|
# Hard limit exceeded triggers immediate restart regardless of strikes.
|
||||||
SOFT_MEM_THRESHOLD=80
|
SOFT_MEM_THRESHOLD=80
|
||||||
|
|
||||||
# HTTP responsiveness check
|
# HTTP responsiveness check settings
|
||||||
RESP_FAIL_LIMIT=2 # consecutive failed curl checks before restart
|
RESP_FAIL_LIMIT=2 # consecutive failed curl checks before restart
|
||||||
CURL_TIMEOUT=5 # seconds before curl gives up per check
|
CURL_TIMEOUT=5 # seconds before curl gives up per check
|
||||||
|
|
||||||
# ━━━ Docker Network Connect ━━━
|
# ━━━ Docker Network Connect ━━━
|
||||||
# Connects containers to extra Docker networks on array start — many-to-many
|
# Connects containers to extra Docker networks on array start.
|
||||||
# Every container in the list connects to every network in the list
|
# Useful when containers need to communicate across networks they were not originally
|
||||||
# Useful when containers need to communicate across networks they weren't configured with
|
# configured with — e.g. memcached needing access to the nextcloud-aio network.
|
||||||
# Comment out entries to disable without removing them
|
# Every container in the list connects to every network in the list (many-to-many).
|
||||||
|
# Comment out entries to disable without removing them.
|
||||||
NETWORK_CONNECT_CONTAINERS=(
|
NETWORK_CONNECT_CONTAINERS=(
|
||||||
"memcached"
|
"memcached"
|
||||||
"Npm-CrowdSec"
|
"Npm-CrowdSec"
|
||||||
)
|
)
|
||||||
|
|
||||||
NETWORK_CONNECT_NETWORKS=(
|
NETWORK_CONNECT_NETWORKS=(
|
||||||
"nextcloud-aio"
|
"nextcloud-aio" # Docker network name — must exist before array start
|
||||||
)
|
)
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
@@ -434,52 +474,56 @@ NETWORK_CONNECT_NETWORKS=(
|
|||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
|
|
||||||
# ━━━ Reboot ━━━
|
# ━━━ Reboot ━━━
|
||||||
# Seconds of warning broadcast to logged-in users before server_reboot.sh reboots
|
# Seconds of warning broadcast to all logged-in users before server_reboot.sh reboots.
|
||||||
|
# Gives users time to save work or finish what they are doing before the system goes down.
|
||||||
REBOOT_SLEEP=300
|
REBOOT_SLEEP=300
|
||||||
|
|
||||||
# ━━━ Mover ━━━
|
# ━━━ Mover ━━━
|
||||||
# Seconds to wait after warning users before mover_stop.sh kills the mover process
|
# Seconds to wait after warning users before mover_stop.sh sends SIGTERM to the mover.
|
||||||
|
# Gives the mover time to finish its current file operation cleanly before being killed.
|
||||||
MOVER_STOP_TIMEOUT=300
|
MOVER_STOP_TIMEOUT=300
|
||||||
|
|
||||||
# ━━━ Syslog Filter ━━━
|
# ━━━ Syslog Filter ━━━
|
||||||
# Path for the rsyslog filter file that suppresses Docker veth/docker0 noise
|
# Path for the rsyslog filter file created by docker_syslog_filter.sh.
|
||||||
|
# The filter suppresses noisy Docker veth and docker0 messages from syslog on boot.
|
||||||
|
# Without this filter, every Docker network interface change floods the syslog.
|
||||||
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
FILTER_FILE="/etc/rsyslog.d/ignore-docker-veth.conf"
|
||||||
|
|
||||||
# ━━━ PHP-FPM ━━━
|
# ━━━ PHP-FPM ━━━
|
||||||
# Config file path and max children value for php_fpm_max_children.sh
|
# Config file path and max children value for php_fpm_max_children.sh.
|
||||||
|
# Higher max_children allows more concurrent PHP requests to the unRAID WebGUI.
|
||||||
|
# Set based on available RAM — too high can cause memory pressure on low-RAM systems.
|
||||||
PHP_CONF="/etc/php-fpm.d/www.conf"
|
PHP_CONF="/etc/php-fpm.d/www.conf"
|
||||||
PHP_MAX_CHILDREN=250
|
PHP_MAX_CHILDREN=250
|
||||||
|
|
||||||
# ━━━ Clear Logs ━━━
|
# ━━━ Clear Logs ━━━
|
||||||
# System log files cleared by clear_logs.sh — Docker logs cleared automatically too
|
# System log files cleared by clear_logs.sh — Docker container logs are cleared too.
|
||||||
|
# Run weekly to prevent logs from filling the rootfs over time.
|
||||||
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
|
LOG_FILES=(/var/log/syslog /var/log/messages /var/log/dmesg)
|
||||||
|
|
||||||
# ━━━ WebGUI Watchdog ━━━
|
# ━━━ WebGUI Watchdog ━━━
|
||||||
# Monitors unRAID WebGUI and restarts services if unresponsive
|
# Monitors the unRAID WebGUI and restarts services if it becomes unresponsive.
|
||||||
# Escalation: nginx restart → recheck → emhttp restart → recheck → notify warning
|
# Escalation path: nginx restart → recheck → emhttp restart → recheck → notify warning.
|
||||||
|
# emhttp is the core unRAID management daemon — restarting it is more disruptive than nginx
|
||||||
|
# but both recover cleanly. Notification sent on any restart so you know what happened.
|
||||||
WEBGUI_URL="http://localhost" # adjust port if non-standard e.g. http://localhost:8080
|
WEBGUI_URL="http://localhost" # adjust port if non-standard e.g. http://localhost:8080
|
||||||
WEBGUI_TIMEOUT=5 # seconds before curl gives up
|
WEBGUI_TIMEOUT=5 # seconds before curl gives up on the WebGUI check
|
||||||
WEBGUI_NGINX_WAIT=15 # seconds to wait after nginx restart before recheck
|
WEBGUI_NGINX_WAIT=15 # seconds to wait after nginx restart before rechecking
|
||||||
WEBGUI_EMHTTP_WAIT=30 # seconds to wait after emhttp restart before recheck
|
WEBGUI_EMHTTP_WAIT=30 # seconds to wait after emhttp restart — emhttp takes longer
|
||||||
|
|
||||||
# ━━━ ZFS Memory Snapshot ━━━
|
|
||||||
# Weekly ZFS pool health and memory diagnostic report — informational only, no action taken
|
|
||||||
# system_watchdog.sh handles threshold-based intervention
|
|
||||||
ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log"
|
|
||||||
ZFS_REPORT_ARC_WARN_PCT=90 # warn if ARC utilization above this %
|
|
||||||
ZFS_REPORT_FREE_WARN_GB=10 # warn if free RAM below this GB
|
|
||||||
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if available RAM below this GB
|
|
||||||
ZFS_REPORT_DOCKER_TOP=10 # number of top Docker memory users to show in report
|
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── MEDIA ─────────────────────────────────────────────────────────────────────────────────────
|
# ── MEDIA ─────────────────────────────────────────────────────────────────────────────────────
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
|
|
||||||
# ━━━ Media Permissions ━━━
|
# ━━━ Media Permissions ━━━
|
||||||
# Mode and owner applied recursively to all shares in MEDIA_PERMISSION_SHARES
|
# Mode and owner applied recursively to all shares in MEDIA_PERMISSION_SHARES.
|
||||||
|
# Run by Media/media_shares_permissions.sh via the media_management.sh orchestrator.
|
||||||
|
# 777 and nobody:users is standard for unRAID media shares accessible by Docker containers.
|
||||||
PERMISSIONS_MODE="777"
|
PERMISSIONS_MODE="777"
|
||||||
PERMISSIONS_OWNER="nobody:users"
|
PERMISSIONS_OWNER="nobody:users"
|
||||||
|
|
||||||
|
# Shares to apply permissions to — add or remove paths as your library grows.
|
||||||
|
# These are applied recursively so large shares take time — run overnight via orchestrator.
|
||||||
MEDIA_PERMISSION_SHARES=(
|
MEDIA_PERMISSION_SHARES=(
|
||||||
/mnt/user/Anime_Movies
|
/mnt/user/Anime_Movies
|
||||||
/mnt/user/Anime_Movies-Old
|
/mnt/user/Anime_Movies-Old
|
||||||
@@ -506,10 +550,12 @@ MEDIA_PERMISSION_SHARES=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# ━━━ Media Cleaner ━━━
|
# ━━━ Media Cleaner ━━━
|
||||||
# Two profiles: anime and media — passed as argument to media_cleaner.sh
|
# Removes junk files from media shares using configurable file pattern lists.
|
||||||
# Run via Orchestrators/media_management.sh for permissions + both cleaners in one job
|
# Two profiles: anime and media — each with their own folder list and patterns.
|
||||||
# Usage: media_cleaner.sh anime or media_cleaner.sh media
|
# Run via Media/media_cleaner.sh anime or Media/media_cleaner.sh media
|
||||||
|
# Called automatically by media_management.sh via MEDIA_MAINTENANCE_JOBS below.
|
||||||
|
|
||||||
|
# Folders scanned by the anime profile — anime downloads commonly include these junk files
|
||||||
ANIME_CLEAN_FOLDERS=(
|
ANIME_CLEAN_FOLDERS=(
|
||||||
/mnt/user/Anime_Movies
|
/mnt/user/Anime_Movies
|
||||||
/mnt/user/Anime_Movies-Old
|
/mnt/user/Anime_Movies-Old
|
||||||
@@ -517,6 +563,7 @@ ANIME_CLEAN_FOLDERS=(
|
|||||||
/mnt/user/Anime_Shows-Old
|
/mnt/user/Anime_Shows-Old
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Folders scanned by the media profile
|
||||||
MEDIA_CLEAN_FOLDERS=(
|
MEDIA_CLEAN_FOLDERS=(
|
||||||
/mnt/user/Kids_Movies
|
/mnt/user/Kids_Movies
|
||||||
/mnt/user/Kids_Tv_Shows
|
/mnt/user/Kids_Tv_Shows
|
||||||
@@ -527,7 +574,7 @@ MEDIA_CLEAN_FOLDERS=(
|
|||||||
/mnt/user/Tv_Shows
|
/mnt/user/Tv_Shows
|
||||||
)
|
)
|
||||||
|
|
||||||
# Junk file patterns common in anime downloads
|
# File patterns deleted by the anime profile — common junk from anime download groups
|
||||||
ANIME_FILE_PATTERNS=(
|
ANIME_FILE_PATTERNS=(
|
||||||
'*.sfv' '*.md5' '*.sha1' '*.txt' '*.url' '*.lnk'
|
'*.sfv' '*.md5' '*.sha1' '*.txt' '*.url' '*.lnk'
|
||||||
'*.rar' '*.zip' '*.info' '*.torrent' '*.sample*' '*.proof*'
|
'*.rar' '*.zip' '*.info' '*.torrent' '*.sample*' '*.proof*'
|
||||||
@@ -535,7 +582,7 @@ ANIME_FILE_PATTERNS=(
|
|||||||
'*.log' '*.json'
|
'*.log' '*.json'
|
||||||
)
|
)
|
||||||
|
|
||||||
# Junk file patterns for general media — includes *.iso and *.lrc not needed in anime
|
# File patterns deleted by the media profile — includes *.iso and *.lrc not needed in anime
|
||||||
MEDIA_FILE_PATTERNS=(
|
MEDIA_FILE_PATTERNS=(
|
||||||
'*.sfv' '*.md5' '*.sha1' '*.txt' '*.url' '*.lnk'
|
'*.sfv' '*.md5' '*.sha1' '*.txt' '*.url' '*.lnk'
|
||||||
'*.rar' '*.zip' '*.info' '*.torrent' '*.sample*' '*.proof*'
|
'*.rar' '*.zip' '*.info' '*.torrent' '*.sample*' '*.proof*'
|
||||||
@@ -544,108 +591,252 @@ MEDIA_FILE_PATTERNS=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# ━━━ Media Management Orchestrator ━━━
|
# ━━━ Media Management Orchestrator ━━━
|
||||||
# Scripts run sequentially by Orchestrators/media_management.sh
|
# Job list for Orchestrators/media_management.sh — runs scripts sequentially in order.
|
||||||
# Order matters — permissions runs first so cleaner operates on correctly owned files
|
# Format: "folder/script.sh optional_argument"
|
||||||
# Format: "script_path [optional_argument]"
|
# Order matters — permissions runs first so cleaners and arr scripts see correct ownership.
|
||||||
# Add a new line to run another script — comment out to disable without removing
|
# Arr cleanup scripts run last — they depend on clean folders from the cleaner steps.
|
||||||
|
# Comment out any job to disable without removing it — easy to re-enable later.
|
||||||
MEDIA_MAINTENANCE_JOBS=(
|
MEDIA_MAINTENANCE_JOBS=(
|
||||||
"Media/media_shares_permissions.sh"
|
"Media/media_shares_permissions.sh" # apply permissions first
|
||||||
"Media/media_cleaner.sh anime"
|
"Media/media_cleaner.sh anime" # remove junk from anime shares
|
||||||
"Media/media_cleaner.sh media"
|
"Media/media_cleaner.sh media" # remove junk from media shares
|
||||||
|
"Media/lidarr_cleanup.sh" # remove orphaned music files
|
||||||
|
"Media/sonarr_cleanup.sh" # remove orphaned TV files
|
||||||
|
"Media/radarr_cleanup.sh" # remove orphaned movie files
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# ━━━ Arr Cleanup ━━━
|
||||||
|
# Lidarr, Sonarr and Radarr orphan file cleanup via their respective APIs.
|
||||||
|
# Each arr script queries its API to get all tracked file paths, then compares against
|
||||||
|
# what exists on disk. Files not tracked by the arr and older than ORPHAN_AGE days are deleted.
|
||||||
|
#
|
||||||
|
# Why the age threshold matters:
|
||||||
|
# The arr downloads a file then processes it — there is a window where the file exists
|
||||||
|
# on disk but the arr hasn't imported it yet. ORPHAN_AGE prevents deleting files that
|
||||||
|
# are mid-import. 7 days is conservative and safe for any normal workflow.
|
||||||
|
#
|
||||||
|
# Protected patterns are NEVER deleted regardless of tracking status or age.
|
||||||
|
# These protect arr-generated metadata (cover art, .nfo files, subtitles) that the arr
|
||||||
|
# depends on but does not include in its tracked file API response.
|
||||||
|
# Add new patterns here if arr metadata formats change in future versions.
|
||||||
|
|
||||||
|
# Lidarr — music library
|
||||||
|
LIDARR_URL="http://192.168.50.2:8686"
|
||||||
|
LIDARR_API_KEY="b2977e71ef074bc0a0529d9fcce3b2dc"
|
||||||
|
LIDARR_MUSIC_ROOT="/mnt/user/Music-New" # must match the root path set in Lidarr
|
||||||
|
LIDARR_ORPHAN_AGE=7 # days before untracked file is eligible for deletion
|
||||||
|
LIDARR_EXTENSIONS=("flac" "mp3" "m4a" "wav" "aac" "ogg" "opus" "wma")
|
||||||
|
# file extensions considered valid music files
|
||||||
|
LIDARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.lrc")
|
||||||
|
# never deleted — cover art, metadata, lyrics
|
||||||
|
|
||||||
|
# Sonarr — TV library
|
||||||
|
SONARR_URL="http://192.168.50.2:8989"
|
||||||
|
SONARR_API_KEY="130decd3db5b4c25afad64864cd03f9f"
|
||||||
|
SONARR_TV_ROOT="/mnt/user/Tv_Shows" # must match the root path set in Sonarr
|
||||||
|
SONARR_ORPHAN_AGE=7
|
||||||
|
SONARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "ts" "wmv" "mov")
|
||||||
|
SONARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa")
|
||||||
|
# never deleted — artwork, metadata, subtitles
|
||||||
|
|
||||||
|
# Radarr — movie library
|
||||||
|
RADARR_URL="http://192.168.50.2:7878"
|
||||||
|
RADARR_API_KEY="d43a3ec6cf1549edb4af0cc63f98b2a9"
|
||||||
|
RADARR_MOVIES_ROOT="/mnt/user/Movies" # must match the root path set in Radarr
|
||||||
|
RADARR_ORPHAN_AGE=7
|
||||||
|
RADARR_EXTENSIONS=("mkv" "mp4" "avi" "m4v" "wmv" "mov")
|
||||||
|
RADARR_PROTECTED_PATTERNS=("*.jpg" "*.jpeg" "*.png" "*.nfo" "*.srt" "*.sub" "*.ass" "*.ssa")
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── TRANSCODES ────────────────────────────────────────────────────────────────────────────────
|
# ── TRANSCODES ────────────────────────────────────────────────────────────────────────────────
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# Session-based storage allocator using filesystem indirection.
|
# Session-based storage allocator using filesystem indirection.
|
||||||
# ffmpeg resolves the symlink path once at session start — existing sessions are never affected.
|
# ffmpeg resolves the symlink path ONCE at session start — existing sessions are never affected
|
||||||
# New sessions land wherever TRANSCODE_LINK points at the moment they start.
|
# by symlink changes. Only NEW sessions care about where the symlink currently points.
|
||||||
#
|
#
|
||||||
# Flow:
|
# How it works:
|
||||||
# ramdisk_setup.sh — run once at array start, creates ramdisk and sets symlink
|
# ramdisk_setup.sh — run once at array start, creates tmpfs and sets symlink
|
||||||
# transcode_manager.sh — every 3 min, monitors usage and flips symlink if needed
|
# transcode_manager.sh — every 3 min, monitors ramdisk usage and flips symlink if needed
|
||||||
# transcode_cleanup.sh — every 5 min, removes old inactive files from both locations
|
# transcode_cleanup.sh — every 5 min, removes old inactive files from both locations
|
||||||
#
|
#
|
||||||
# Hysteresis gap between RAMDISK_WARN_GB and RAMDISK_LOW_GB prevents flip-flop
|
# Hysteresis gap between RAMDISK_WARN_GB and RAMDISK_LOW_GB prevents flip-flop
|
||||||
# when usage hovers near the threshold — gap should be at least 0.5-1GB
|
# when usage hovers near the threshold. Gap should be at least 0.5-1GB.
|
||||||
|
# Ramdisk is dynamic tmpfs — only uses RAM actually needed, RAMDISK_SIZE is the ceiling.
|
||||||
|
|
||||||
RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point — created at array start
|
RAMDISK_PATH="/mnt/ramdisk_transcodes" # tmpfs mount point created at array start
|
||||||
RAMDISK_SIZE="8G" # ceiling — tmpfs is dynamic, only uses what's needed
|
RAMDISK_SIZE="8G" # ceiling — tmpfs only uses RAM actually needed
|
||||||
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at — never changes location
|
TRANSCODE_LINK="/mnt/ram-transcode" # symlink Emby points at — location never changes
|
||||||
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback for edge cases
|
TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/" # SSD fallback for edge cases only
|
||||||
|
|
||||||
# Thresholds in GB — flip symlink to SSD at WARN, flip back to ramdisk at LOW
|
# Usage thresholds in GB
|
||||||
RAMDISK_WARN_GB=6.8 # flip to SSD at or above this — "getting full"
|
RAMDISK_WARN_GB=6.8 # flip symlink to SSD at or above this usage
|
||||||
RAMDISK_LOW_GB=5.5 # flip back to ramdisk when cleanup brings usage here
|
RAMDISK_LOW_GB=5.5 # flip symlink back to ramdisk when usage drops here
|
||||||
RAMDISK_SSD_MIN_GB=20 # minimum free GB on SSD before allowing flip — safety net
|
RAMDISK_SSD_MIN_GB=20 # minimum free GB on SSD required before allowing flip to SSD
|
||||||
|
|
||||||
# Cleanup — files must be older than MAX_AGE and not open by any process to be deleted
|
# Cleanup age thresholds — files must be older than these AND not open by any process
|
||||||
TRANSCODE_MAX_AGE=20 # minutes before a file is eligible for cleanup
|
TRANSCODE_MAX_AGE=20 # minutes before a transcode file is eligible for cleanup
|
||||||
TRANSCODE_ORPHAN_AGE=30 # minutes before an orphaned file is eligible — extra caution
|
TRANSCODE_ORPHAN_AGE=30 # minutes before an orphaned file is eligible — extra caution buffer
|
||||||
|
|
||||||
# Flip frequency monitoring — notify if symlink flips too often (indicates sizing issue)
|
# Flip frequency alert — too many flips per hour may indicate ramdisk needs to be larger
|
||||||
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times in one hour
|
TRANSCODE_FLIP_WARN=3 # notify if symlink flips this many times in one hour
|
||||||
|
|
||||||
# Permissions — must match your Emby container user
|
# Permissions applied to ramdisk and SSD fallback — must match your Emby container user
|
||||||
TRANSCODE_OWNER="nobody:users"
|
TRANSCODE_OWNER="nobody:users"
|
||||||
TRANSCODE_MODE="755"
|
TRANSCODE_MODE="755"
|
||||||
|
|
||||||
|
# ==============================================================================================
|
||||||
|
# ── MONITOR ───────────────────────────────────────────────────────────────────────────────────
|
||||||
|
# ==============================================================================================
|
||||||
|
# Monitoring scripts — watch and report only, never take action.
|
||||||
|
# Lives in Monitor/ folder — distinct from unRAID_Essentials (which acts) and
|
||||||
|
# Docker_Essentials (which manages containers).
|
||||||
|
# These scripts are the data sources for the future plugin dashboard.
|
||||||
|
|
||||||
|
# ━━━ Certificate Monitor ━━━
|
||||||
|
# Checks SSL cert expiry via direct openssl connection — no NPM dependency.
|
||||||
|
# Reads the actual cert the server is presenting — catches real-world issues API checks miss.
|
||||||
|
# Each domain and subdomain is a separate entry — they have independent certs.
|
||||||
|
# Add your public-facing domains — uncomment and replace with your actual domains.
|
||||||
|
CERT_MONITOR_DOMAINS=(
|
||||||
|
# "yourdomain.com"
|
||||||
|
# "auth.yourdomain.com"
|
||||||
|
# "emby.yourdomain.com"
|
||||||
|
# "nextcloud.yourdomain.com"
|
||||||
|
)
|
||||||
|
CERT_WARN_DAYS=30 # notify warning when cert expires within this many days
|
||||||
|
CERT_CRIT_DAYS=7 # notify critical when cert expires within this many days
|
||||||
|
CERT_TIMEOUT=10 # seconds before openssl connection attempt gives up per domain
|
||||||
|
|
||||||
|
# ━━━ Backup Verify ━━━
|
||||||
|
# Verifies the rsync mirror is healthy by comparing random file checksums between servers.
|
||||||
|
# Uses existing SSH keys — no additional configuration needed beyond the share list.
|
||||||
|
# Leave BACKUP_VERIFY_SHARES empty to automatically use DAILY_SYNC_SHARES as the target list.
|
||||||
|
BACKUP_VERIFY_SHARES=(
|
||||||
|
# leave empty to use DAILY_SYNC_SHARES automatically
|
||||||
|
# or specify individual shares to verify:
|
||||||
|
# /mnt/user/Movies
|
||||||
|
# /mnt/user/Tv_Shows
|
||||||
|
)
|
||||||
|
BACKUP_VERIFY_SAMPLE=10 # number of files to randomly sample per share per run
|
||||||
|
BACKUP_VERIFY_MIN_SIZE=1M # skip files smaller than this — avoids tiny junk files
|
||||||
|
|
||||||
|
# ━━━ SMART Health ━━━
|
||||||
|
# Monitors drive SMART attributes — reads live from each drive, no persistent writes.
|
||||||
|
# Discovers all drives automatically via /dev/sd* and /dev/nvme* — no drive list needed.
|
||||||
|
# Add drives to SMART_IGNORE_DRIVES to skip specific drives (e.g. your unRAID boot USB).
|
||||||
|
SMART_TEMP_WARN=45 # degrees C — warn if drive temperature exceeds this
|
||||||
|
SMART_TEMP_CRIT=55 # degrees C — critical if drive temperature exceeds this
|
||||||
|
SMART_IGNORE_DRIVES=(
|
||||||
|
# "sda" # uncomment to ignore sda — common choice if sda is your unRAID boot USB
|
||||||
|
)
|
||||||
|
|
||||||
|
# ━━━ Bandwidth Monitor ━━━
|
||||||
|
# Logs daily rsync transfer totals to a bounded file on /boot/ — minimal flash wear.
|
||||||
|
# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each successful sync.
|
||||||
|
# bandwidth_monitor.sh --report generates the weekly summary standalone.
|
||||||
|
# File stays bounded to BANDWIDTH_LOG_RETENTION lines — old entries auto-purged on each write.
|
||||||
|
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
|
||||||
|
BANDWIDTH_LOG_RETENTION=90 # days to keep — file never grows beyond ~90 lines
|
||||||
|
BANDWIDTH_WARN_GB=50 # flag in reports if a single sync transfer exceeds this GB
|
||||||
|
|
||||||
|
# ━━━ Health Digest ━━━
|
||||||
|
# Aggregated system health summary from across the ecosystem.
|
||||||
|
# Reads existing state files — no new writes to flash drive.
|
||||||
|
#
|
||||||
|
# Three profiles — switch by changing DIGEST_PROFILE, no cron changes needed:
|
||||||
|
# always — sends every run (schedule daily = daily digest, weekly = weekly digest)
|
||||||
|
# smart — sends only if findings worth reporting (intelligent quiet operation)
|
||||||
|
# weekly — sends once per week on DIGEST_DAY only, silent all other days
|
||||||
|
#
|
||||||
|
# Data sources (reads only — no writes):
|
||||||
|
# Transcode ramdisk state, container watchdog strikes, system watchdog strikes,
|
||||||
|
# failover state, container skip list, bandwidth history, SSL cert days remaining
|
||||||
|
DIGEST_PROFILE="weekly" # always | smart | weekly
|
||||||
|
DIGEST_DAY="Sunday" # day name for weekly profile — must match date +%A output
|
||||||
|
|
||||||
|
# Smart profile triggers — set true to send digest when this condition is found
|
||||||
|
DIGEST_SMART_ON_WATCHDOG=true # send if any watchdog strikes are active
|
||||||
|
DIGEST_SMART_ON_FAILOVER=true # send if failover state is not NORMAL
|
||||||
|
DIGEST_SMART_ON_CERT_WARN=true # send if any cert is under CERT_WARN_DAYS
|
||||||
|
DIGEST_SMART_ON_BANDWIDTH=true # send if any transfer exceeded BANDWIDTH_WARN_GB
|
||||||
|
|
||||||
|
# ━━━ Emby Session Report ━━━
|
||||||
|
# Weekly Emby usage report via API — no persistent writes, queries fresh each run.
|
||||||
|
# Shows active streams, library counts, transcode vs direct play ratio.
|
||||||
|
# Requires an API key from Emby Settings → API Keys in the Emby WebUI.
|
||||||
|
EMBY_URL="http://localhost:8096"
|
||||||
|
EMBY_API_KEY="" # paste your Emby API key here
|
||||||
|
EMBY_REPORT_DAYS=7 # number of days to include in the report period
|
||||||
|
EMBY_REPORT_TOP_N=10 # number of top content items to show in report
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# Last line of defense — reboots cleanly if the system is about to become unstable.
|
# Last line of defense — reboots the system cleanly when it is about to become unstable.
|
||||||
# Works alongside docker_watchdog.sh — containers first, system reboot second.
|
# Runs every 15 minutes via cron. Works alongside docker_watchdog.sh:
|
||||||
# Thresholds set at "about to fall over" levels — not just high usage.
|
# docker_watchdog.sh — container level, minimal disruption, tries to self-heal first
|
||||||
# Strike system prevents rebooting on single spikes — sustained threshold hits trigger action.
|
# system_watchdog.sh — system level, last resort, reboots when healing has failed
|
||||||
|
#
|
||||||
|
# Strike system: sustained threshold hits trigger reboot — single spikes are ignored.
|
||||||
|
# Each check that exceeds its threshold adds a strike. Strikes reset when recovered.
|
||||||
|
# When strike limit is hit the reboot sequence begins.
|
||||||
|
#
|
||||||
|
# Reboot loop protection: tracks reboot timestamps on /boot/ (survives reboots).
|
||||||
|
# If the server reboots too many times in the window it shuts down instead — a reboot
|
||||||
|
# loop means something is fundamentally wrong that a reboot is not fixing.
|
||||||
|
|
||||||
# ━━━ State Files ━━━
|
# ━━━ State Files ━━━
|
||||||
# Strike counts reset on reboot — /tmp is correct for this
|
# Strike counts reset on reboot — /tmp is correct (fresh start after each reboot)
|
||||||
SYS_WATCHDOG_STATE_FILE="/tmp/system_watchdog_state.db"
|
SYS_WATCHDOG_STATE_FILE="/tmp/system_watchdog_state.db"
|
||||||
# Persistent container skip list — survives reboots, auto-clears when container recovers
|
# Persistent container skip list — on /boot/ so it survives reboots
|
||||||
|
# Containers added here when docker_watchdog.sh exhausts all restart attempts
|
||||||
|
# Auto-clears when container is found running again after reboot or manual fix
|
||||||
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
|
||||||
# Reboot timestamp log — survives reboots for loop detection
|
# Reboot timestamp log — on /boot/ for reboot loop detection across reboots
|
||||||
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
SYS_WATCHDOG_REBOOT_LOG="/boot/config/system_watchdog_reboots.db"
|
||||||
|
|
||||||
# ━━━ Strike and Reboot Loop Settings ━━━
|
# ━━━ Strike and Reboot Loop Settings ━━━
|
||||||
# Consecutive threshold hits before triggering reboot
|
# Consecutive threshold hits required before triggering reboot
|
||||||
SYS_WATCHDOG_STRIKE_LIMIT=2
|
SYS_WATCHDOG_STRIKE_LIMIT=2
|
||||||
# Maximum reboots allowed in window before shutdown instead — prevents reboot loops
|
# Maximum reboots allowed within the window before shutting down instead
|
||||||
|
# A reboot loop means something fundamental is broken that rebooting is not fixing
|
||||||
SYS_WATCHDOG_REBOOT_LIMIT=3
|
SYS_WATCHDOG_REBOOT_LIMIT=3
|
||||||
# Window in hours — controls both reboot count window AND rolling log purge
|
# Window in hours — controls BOTH the reboot count window AND the rolling log purge
|
||||||
# 12 = entries older than 12hrs purge automatically, fresh window starts
|
# Entries older than this many hours are automatically removed from the reboot log
|
||||||
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
|
SYS_WATCHDOG_REBOOT_WINDOW_HRS=12
|
||||||
|
|
||||||
# ━━━ Thresholds ━━━
|
# ━━━ Thresholds ━━━
|
||||||
# Set these at "I am about to become unstable" levels — not just "things are a bit tight"
|
# Set these at "I am about to become unstable" levels — not just "things are a bit high"
|
||||||
SYS_WATCHDOG_ROOTFS_PCT=95 # rootfs % — at 95% something is seriously wrong
|
SYS_WATCHDOG_ROOTFS_PCT=95 # rootfs % — at 95% something is seriously wrong
|
||||||
SYS_WATCHDOG_LOG_PCT=95 # /var/log % — log spam filling disk
|
SYS_WATCHDOG_LOG_PCT=95 # /var/log % — log spam filling the filesystem
|
||||||
SYS_WATCHDOG_MEM_GB=4 # free RAM GB — 4GB on 128GB system is critical
|
SYS_WATCHDOG_MEM_GB=4 # free RAM GB — 4GB free on 128GB system is critical
|
||||||
SYS_WATCHDOG_ARC_PINNED_PCT=98 # ZFS ARC % of max before attempting reclaim
|
SYS_WATCHDOG_ARC_PINNED_PCT=98 # ZFS ARC % of max before attempting reclaim
|
||||||
SYS_WATCHDOG_ARC_RELEASE_PCT=95 # ZFS ARC % after reclaim that still triggers reboot
|
SYS_WATCHDOG_ARC_RELEASE_PCT=95 # ZFS ARC % after reclaim that still triggers reboot
|
||||||
SYS_WATCHDOG_LOAD_MULTIPLIER=3 # strike if load avg > cores x this value
|
SYS_WATCHDOG_LOAD_MULTIPLIER=3 # strike if load avg > cores x this multiplier
|
||||||
SYS_WATCHDOG_ZOMBIE_LIMIT=50 # zombie process count before strike
|
SYS_WATCHDOG_ZOMBIE_LIMIT=50 # zombie process count before strike
|
||||||
SYS_WATCHDOG_CPU_TEMP_MAX=95 # degrees C — adjust for your CPU tjmax
|
SYS_WATCHDOG_CPU_TEMP_MAX=95 # degrees C — adjust for your specific CPU tjmax
|
||||||
|
|
||||||
# ━━━ Check Toggles ━━━
|
# ━━━ Check Toggles ━━━
|
||||||
# true = run this check / false = skip entirely
|
# true = run this check on every watchdog cycle / false = skip entirely
|
||||||
# Disable checks that are not relevant to your hardware or cause false positives
|
# Disable checks not relevant to your hardware or that cause false positives
|
||||||
SYS_WATCHDOG_CHECK_ROOTFS=true
|
SYS_WATCHDOG_CHECK_ROOTFS=true
|
||||||
SYS_WATCHDOG_CHECK_LOG=true
|
SYS_WATCHDOG_CHECK_LOG=true
|
||||||
SYS_WATCHDOG_CHECK_RAM=true
|
SYS_WATCHDOG_CHECK_RAM=true
|
||||||
SYS_WATCHDOG_CHECK_ARC=true
|
SYS_WATCHDOG_CHECK_ARC=true
|
||||||
SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
||||||
SYS_WATCHDOG_CHECK_LOAD=false
|
SYS_WATCHDOG_CHECK_LOAD=false # disabled — load spikes during transcoding are normal
|
||||||
SYS_WATCHDOG_CHECK_ZOMBIES=true
|
SYS_WATCHDOG_CHECK_ZOMBIES=true
|
||||||
SYS_WATCHDOG_CHECK_CONTAINERS=true
|
SYS_WATCHDOG_CHECK_CONTAINERS=true # checks persistent skip list from docker_watchdog.sh
|
||||||
SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
||||||
|
|
||||||
# ━━━ Abort Toggles ━━━
|
# ━━━ Abort Toggles ━━━
|
||||||
# true = abort reboot if condition is active / false = reboot anyway
|
# Controls whether certain conditions prevent a reboot from happening.
|
||||||
# Default true = conservative — set false only when "reboot no matter what" is wanted
|
# true = abort reboot if this condition is active (conservative — default)
|
||||||
# Goal: graceful reboot before crash is always better than hard crash mid-operation
|
# false = reboot anyway regardless of this condition (aggressive)
|
||||||
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # unhealthy pool + reboot = potential data loss
|
# Philosophy: a graceful reboot before crash is always better than a hard crash mid-operation
|
||||||
SYS_WATCHDOG_ABORT_ON_PARITY=true # aborting parity better than crashing mid-check
|
SYS_WATCHDOG_ABORT_ON_ZFS_UNHEALTHY=true # unhealthy pool + reboot risks data loss
|
||||||
SYS_WATCHDOG_ABORT_ON_MOVER=true # aborting move better than crashing mid-move
|
SYS_WATCHDOG_ABORT_ON_PARITY=true # aborting parity check beats crashing mid-check
|
||||||
|
SYS_WATCHDOG_ABORT_ON_MOVER=true # aborting mover beats crashing mid-move
|
||||||
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# ──────────────────────── End Of User Variables ───────────────────────────────────────────────
|
# ──────────────────────── End Of User Variables ───────────────────────────────────────────────
|
||||||
|
|||||||
@@ -0,0 +1,291 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- Lidarr Cleanup Script --------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Removes orphaned music files from the library that Lidarr no longer tracks.
|
||||||
|
# Uses the Lidarr API to build a complete list of tracked file paths then compares
|
||||||
|
# against what exists on disk — anything not tracked and older than LIDARR_ORPHAN_AGE
|
||||||
|
# days is considered an orphan and deleted.
|
||||||
|
#
|
||||||
|
# File classification:
|
||||||
|
# TRACKED — Lidarr API knows about this exact file path → leave it alone
|
||||||
|
# PROTECTED — matches LIDARR_PROTECTED_PATTERNS → never delete (cover art, .nfo etc.)
|
||||||
|
# ORPHAN — music file, not tracked, older than LIDARR_ORPHAN_AGE days → delete
|
||||||
|
# JUNK — not a music extension, not protected → delete regardless of age
|
||||||
|
# RECENT — not tracked, under LIDARR_ORPHAN_AGE days old → skip (may be mid-import)
|
||||||
|
#
|
||||||
|
# Why protected patterns matter:
|
||||||
|
# Lidarr generates cover art (*.jpg), metadata (*.nfo) and lyrics (*.lrc) but does
|
||||||
|
# not include these in its tracked file API response. Without protection these would
|
||||||
|
# be classified as orphans and deleted — breaking Lidarr and Emby metadata display.
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under Arr Cleanup section.
|
||||||
|
# Supports --dry-run to preview what would be deleted without making changes.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_GEAR Setup ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||||
|
|
||||||
|
if [[ "$EUID" -ne 0 ]]; then
|
||||||
|
error "Must be run as root"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Running as root"
|
||||||
|
|
||||||
|
if ! command -v curl >/dev/null 2>&1; then
|
||||||
|
error "curl not found — required for Lidarr API calls"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v jq >/dev/null 2>&1; then
|
||||||
|
error "jq not found — required for JSON parsing"
|
||||||
|
notify "Lidarr cleanup failed on $(hostname) — jq not installed" "Lidarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
require_var LIDARR_URL
|
||||||
|
require_var LIDARR_API_KEY
|
||||||
|
require_var LIDARR_MUSIC_ROOT
|
||||||
|
|
||||||
|
if [[ ! -d "$LIDARR_MUSIC_ROOT" ]]; then
|
||||||
|
error "Music root not found: $LIDARR_MUSIC_ROOT"
|
||||||
|
notify "Lidarr cleanup failed on $(hostname) — music root not found: $LIDARR_MUSIC_ROOT" "Lidarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Config validated"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Status ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
if [[ "$SHOW_STATUS" == true ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||||
|
echo "$ICON_GEAR Lidarr URL: $LIDARR_URL"
|
||||||
|
echo "$ICON_GEAR Music root: $LIDARR_MUSIC_ROOT"
|
||||||
|
echo "$ICON_TIME Orphan age: ${LIDARR_ORPHAN_AGE} days"
|
||||||
|
echo "$ICON_GEAR Extensions: ${LIDARR_EXTENSIONS[*]}"
|
||||||
|
echo "$ICON_GEAR Protected patterns: ${LIDARR_PROTECTED_PATTERNS[*]}"
|
||||||
|
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be deleted"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# HELPERS
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Makes a GET request to the Lidarr API — exits on connection failure
|
||||||
|
lidarr_api() {
|
||||||
|
local endpoint="$1"
|
||||||
|
local response http_code body
|
||||||
|
|
||||||
|
response=$(curl -sf \
|
||||||
|
--max-time 30 \
|
||||||
|
-H "X-Api-Key: $LIDARR_API_KEY" \
|
||||||
|
-w "\n%{http_code}" \
|
||||||
|
"${LIDARR_URL}/api/v1/${endpoint}" 2>/dev/null)
|
||||||
|
|
||||||
|
http_code=$(echo "$response" | tail -1)
|
||||||
|
body=$(echo "$response" | head -n -1)
|
||||||
|
|
||||||
|
if [[ "$http_code" != "200" ]]; then
|
||||||
|
error "Lidarr API returned HTTP $http_code for endpoint: $endpoint"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$body"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Returns 0 if file extension matches LIDARR_EXTENSIONS list
|
||||||
|
is_music_file() {
|
||||||
|
local ext="${1##*.}"
|
||||||
|
ext="${ext,,}"
|
||||||
|
for valid_ext in "${LIDARR_EXTENSIONS[@]}"; do
|
||||||
|
[[ "$ext" == "$valid_ext" ]] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# Returns 0 if filename matches any pattern in LIDARR_PROTECTED_PATTERNS
|
||||||
|
is_protected_file() {
|
||||||
|
local filename
|
||||||
|
filename=$(basename "$1")
|
||||||
|
for pattern in "${LIDARR_PROTECTED_PATTERNS[@]}"; do
|
||||||
|
# shellcheck disable=SC2254
|
||||||
|
case "$filename" in
|
||||||
|
$pattern) return 0 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SYNC Fetching Lidarr Tracked Files ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_SYNC Fetching Lidarr Tracked Files ━━━"
|
||||||
|
|
||||||
|
info "Querying Lidarr API: $LIDARR_URL"
|
||||||
|
|
||||||
|
TRACKFILE_RESPONSE=$(lidarr_api "trackfile") || {
|
||||||
|
error "Failed to fetch track files from Lidarr — check URL and API key"
|
||||||
|
notify "Lidarr cleanup failed on $(hostname) — API unreachable" "Lidarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
TMP_DIR="/tmp/lidarr_cleanup_$$"
|
||||||
|
mkdir -p "$TMP_DIR"
|
||||||
|
trap "rm -rf $TMP_DIR" EXIT
|
||||||
|
|
||||||
|
TRACKED_FILE="$TMP_DIR/tracked_paths.txt"
|
||||||
|
echo "$TRACKFILE_RESPONSE" | jq -r '.[].path' 2>/dev/null | sort > "$TRACKED_FILE"
|
||||||
|
|
||||||
|
TRACKED_COUNT=$(wc -l < "$TRACKED_FILE")
|
||||||
|
success "Lidarr tracks $TRACKED_COUNT files"
|
||||||
|
|
||||||
|
if [[ "$TRACKED_COUNT" -eq 0 ]]; then
|
||||||
|
warn "No tracked files returned — Lidarr may not have scanned yet or library is empty"
|
||||||
|
warn "Aborting to prevent mass deletion"
|
||||||
|
notify "Lidarr cleanup aborted on $(hostname) — no tracked files returned from API" "Lidarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_CLEAN Scanning Music Root ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_CLEAN Scanning Music Root ━━━"
|
||||||
|
info "Root: $LIDARR_MUSIC_ROOT"
|
||||||
|
info "Orphan age: ${LIDARR_ORPHAN_AGE} days"
|
||||||
|
info "Protected: ${LIDARR_PROTECTED_PATTERNS[*]}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
START=$(date +%s)
|
||||||
|
ORPHAN_COUNT=0
|
||||||
|
JUNK_COUNT=0
|
||||||
|
RECENT_COUNT=0
|
||||||
|
PROTECTED_COUNT=0
|
||||||
|
ORPHAN_BYTES=0
|
||||||
|
JUNK_BYTES=0
|
||||||
|
|
||||||
|
AGE_SECONDS=$(( LIDARR_ORPHAN_AGE * 86400 ))
|
||||||
|
NOW=$(date +%s)
|
||||||
|
|
||||||
|
while IFS= read -r filepath; do
|
||||||
|
[[ -z "$filepath" ]] && continue
|
||||||
|
|
||||||
|
# TRACKED — Lidarr knows about this file
|
||||||
|
if grep -qF "$filepath" "$TRACKED_FILE" 2>/dev/null; then
|
||||||
|
log "TRACKED: $filepath"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# PROTECTED — never delete regardless of tracking status
|
||||||
|
if is_protected_file "$filepath"; then
|
||||||
|
log "PROTECTED: $filepath"
|
||||||
|
((PROTECTED_COUNT++))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
FILE_SIZE=$(stat -c%s "$filepath" 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
if is_music_file "$filepath"; then
|
||||||
|
# Music file not tracked — check age
|
||||||
|
FILE_MTIME=$(stat -c %Y "$filepath" 2>/dev/null || echo 0)
|
||||||
|
FILE_AGE=$(( NOW - FILE_MTIME ))
|
||||||
|
|
||||||
|
if [[ "$FILE_AGE" -lt "$AGE_SECONDS" ]]; then
|
||||||
|
log "RECENT (skipping): $filepath"
|
||||||
|
((RECENT_COUNT++))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ORPHAN — old enough to delete
|
||||||
|
warn "$ICON_TRASH ORPHAN: $filepath"
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
rm -f "$filepath" && {
|
||||||
|
((ORPHAN_COUNT++))
|
||||||
|
ORPHAN_BYTES=$((ORPHAN_BYTES + FILE_SIZE))
|
||||||
|
} || error "Failed to delete: $filepath"
|
||||||
|
else
|
||||||
|
((ORPHAN_COUNT++))
|
||||||
|
ORPHAN_BYTES=$((ORPHAN_BYTES + FILE_SIZE))
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
# JUNK — not a music extension, not protected
|
||||||
|
log "JUNK: $filepath"
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
rm -f "$filepath" && {
|
||||||
|
((JUNK_COUNT++))
|
||||||
|
JUNK_BYTES=$((JUNK_BYTES + FILE_SIZE))
|
||||||
|
} || error "Failed to delete: $filepath"
|
||||||
|
else
|
||||||
|
((JUNK_COUNT++))
|
||||||
|
JUNK_BYTES=$((JUNK_BYTES + FILE_SIZE))
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
done < <(find "$LIDARR_MUSIC_ROOT" -type f 2>/dev/null)
|
||||||
|
|
||||||
|
# Empty folder cleanup
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
echo ""
|
||||||
|
info "Cleaning up empty folders..."
|
||||||
|
find "$LIDARR_MUSIC_ROOT" -mindepth 1 -type d -empty -delete 2>/dev/null
|
||||||
|
success "Empty folders removed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
END=$(date +%s)
|
||||||
|
|
||||||
|
# Format bytes helper
|
||||||
|
format_bytes() {
|
||||||
|
local bytes=$1
|
||||||
|
if (( bytes > 1073741824 )); then
|
||||||
|
awk "BEGIN {printf \"%.1fGB\", $bytes / 1073741824}"
|
||||||
|
elif (( bytes > 1048576 )); then
|
||||||
|
awk "BEGIN {printf \"%.1fMB\", $bytes / 1048576}"
|
||||||
|
else
|
||||||
|
echo "${bytes}B"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
ORPHAN_HUMAN=$(format_bytes $ORPHAN_BYTES)
|
||||||
|
JUNK_HUMAN=$(format_bytes $JUNK_BYTES)
|
||||||
|
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY LIDARR CLEANUP SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_SYNC Tracked by Lidarr: $TRACKED_COUNT files"
|
||||||
|
echo "$ICON_SHIELD Protected: $PROTECTED_COUNT files (cover art, metadata)"
|
||||||
|
echo "$ICON_TRASH Orphans removed: $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
|
||||||
|
echo "$ICON_TRASH Junk removed: $JUNK_COUNT files ($JUNK_HUMAN)"
|
||||||
|
echo "$ICON_TIME Recent skipped: $RECENT_COUNT files (under ${LIDARR_ORPHAN_AGE} days)"
|
||||||
|
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||||
|
echo ""
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "$ICON_WARN Status: DRY RUN — no files deleted"
|
||||||
|
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS CLEAN — nothing to remove"
|
||||||
|
notify "Lidarr cleanup complete on $(hostname) — library is clean" "Lidarr Cleanup" "normal"
|
||||||
|
else
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS DONE — $TOTAL_REMOVED files removed"
|
||||||
|
notify "Lidarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" "Lidarr Cleanup" "normal"
|
||||||
|
fi
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
@@ -0,0 +1,281 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- Sonarr Cleanup Script --------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Removes orphaned TV episode files from the library that Sonarr no longer tracks.
|
||||||
|
# Uses the Sonarr API to build a complete list of tracked episode file paths then compares
|
||||||
|
# against what exists on disk — anything not tracked and older than SONARR_ORPHAN_AGE
|
||||||
|
# days is considered an orphan and deleted.
|
||||||
|
#
|
||||||
|
# File classification:
|
||||||
|
# TRACKED — Sonarr API knows about this exact file path → leave it alone
|
||||||
|
# PROTECTED — matches SONARR_PROTECTED_PATTERNS → never delete (artwork, subtitles, .nfo)
|
||||||
|
# ORPHAN — video file, not tracked, older than SONARR_ORPHAN_AGE days → delete
|
||||||
|
# JUNK — not a video extension, not protected → delete regardless of age
|
||||||
|
# RECENT — not tracked, under SONARR_ORPHAN_AGE days old → skip (may be mid-import)
|
||||||
|
#
|
||||||
|
# Why protected patterns matter:
|
||||||
|
# Sonarr generates show artwork (*.jpg), metadata (*.nfo) and manages subtitles
|
||||||
|
# (*.srt, *.sub, *.ass) but does not include these in its tracked file API response.
|
||||||
|
# Without protection these would be classified as orphans and deleted.
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under Arr Cleanup section.
|
||||||
|
# Supports --dry-run to preview what would be deleted without making changes.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_GEAR Setup ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||||
|
|
||||||
|
if [[ "$EUID" -ne 0 ]]; then
|
||||||
|
error "Must be run as root"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Running as root"
|
||||||
|
|
||||||
|
if ! command -v curl >/dev/null 2>&1; then
|
||||||
|
error "curl not found — required for Sonarr API calls"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! command -v jq >/dev/null 2>&1; then
|
||||||
|
error "jq not found — required for JSON parsing"
|
||||||
|
notify "Sonarr cleanup failed on $(hostname) — jq not installed" "Sonarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
require_var SONARR_URL
|
||||||
|
require_var SONARR_API_KEY
|
||||||
|
require_var SONARR_TV_ROOT
|
||||||
|
|
||||||
|
if [[ ! -d "$SONARR_TV_ROOT" ]]; then
|
||||||
|
error "TV root not found: $SONARR_TV_ROOT"
|
||||||
|
notify "Sonarr cleanup failed on $(hostname) — TV root not found: $SONARR_TV_ROOT" "Sonarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Config validated"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Status ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
if [[ "$SHOW_STATUS" == true ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||||
|
echo "$ICON_GEAR Sonarr URL: $SONARR_URL"
|
||||||
|
echo "$ICON_GEAR TV root: $SONARR_TV_ROOT"
|
||||||
|
echo "$ICON_TIME Orphan age: ${SONARR_ORPHAN_AGE} days"
|
||||||
|
echo "$ICON_GEAR Extensions: ${SONARR_EXTENSIONS[*]}"
|
||||||
|
echo "$ICON_GEAR Protected patterns: ${SONARR_PROTECTED_PATTERNS[*]}"
|
||||||
|
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no files will be deleted"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# HELPERS
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
sonarr_api() {
|
||||||
|
local endpoint="$1"
|
||||||
|
local response http_code body
|
||||||
|
|
||||||
|
response=$(curl -sf \
|
||||||
|
--max-time 30 \
|
||||||
|
-H "X-Api-Key: $SONARR_API_KEY" \
|
||||||
|
-w "\n%{http_code}" \
|
||||||
|
"${SONARR_URL}/api/v3/${endpoint}" 2>/dev/null)
|
||||||
|
|
||||||
|
http_code=$(echo "$response" | tail -1)
|
||||||
|
body=$(echo "$response" | head -n -1)
|
||||||
|
|
||||||
|
if [[ "$http_code" != "200" ]]; then
|
||||||
|
error "Sonarr API returned HTTP $http_code for endpoint: $endpoint"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$body"
|
||||||
|
}
|
||||||
|
|
||||||
|
is_video_file() {
|
||||||
|
local ext="${1##*.}"
|
||||||
|
ext="${ext,,}"
|
||||||
|
for valid_ext in "${SONARR_EXTENSIONS[@]}"; do
|
||||||
|
[[ "$ext" == "$valid_ext" ]] && return 0
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
is_protected_file() {
|
||||||
|
local filename
|
||||||
|
filename=$(basename "$1")
|
||||||
|
for pattern in "${SONARR_PROTECTED_PATTERNS[@]}"; do
|
||||||
|
# shellcheck disable=SC2254
|
||||||
|
case "$filename" in
|
||||||
|
$pattern) return 0 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SYNC Fetching Sonarr Tracked Files ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_SYNC Fetching Sonarr Tracked Files ━━━"
|
||||||
|
|
||||||
|
info "Querying Sonarr API: $SONARR_URL"
|
||||||
|
|
||||||
|
EPISODEFILE_RESPONSE=$(sonarr_api "episodefile") || {
|
||||||
|
error "Failed to fetch episode files from Sonarr — check URL and API key"
|
||||||
|
notify "Sonarr cleanup failed on $(hostname) — API unreachable" "Sonarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
TMP_DIR="/tmp/sonarr_cleanup_$$"
|
||||||
|
mkdir -p "$TMP_DIR"
|
||||||
|
trap "rm -rf $TMP_DIR" EXIT
|
||||||
|
|
||||||
|
TRACKED_FILE="$TMP_DIR/tracked_paths.txt"
|
||||||
|
echo "$EPISODEFILE_RESPONSE" | jq -r '.[].path' 2>/dev/null | sort > "$TRACKED_FILE"
|
||||||
|
|
||||||
|
TRACKED_COUNT=$(wc -l < "$TRACKED_FILE")
|
||||||
|
success "Sonarr tracks $TRACKED_COUNT episode files"
|
||||||
|
|
||||||
|
if [[ "$TRACKED_COUNT" -eq 0 ]]; then
|
||||||
|
warn "No tracked files returned — Sonarr may not have scanned yet or library is empty"
|
||||||
|
warn "Aborting to prevent mass deletion"
|
||||||
|
notify "Sonarr cleanup aborted on $(hostname) — no tracked files returned from API" "Sonarr Cleanup" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_CLEAN Scanning TV Root ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_CLEAN Scanning TV Root ━━━"
|
||||||
|
info "Root: $SONARR_TV_ROOT"
|
||||||
|
info "Orphan age: ${SONARR_ORPHAN_AGE} days"
|
||||||
|
info "Protected: ${SONARR_PROTECTED_PATTERNS[*]}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
START=$(date +%s)
|
||||||
|
ORPHAN_COUNT=0
|
||||||
|
JUNK_COUNT=0
|
||||||
|
RECENT_COUNT=0
|
||||||
|
PROTECTED_COUNT=0
|
||||||
|
ORPHAN_BYTES=0
|
||||||
|
JUNK_BYTES=0
|
||||||
|
|
||||||
|
AGE_SECONDS=$(( SONARR_ORPHAN_AGE * 86400 ))
|
||||||
|
NOW=$(date +%s)
|
||||||
|
|
||||||
|
while IFS= read -r filepath; do
|
||||||
|
[[ -z "$filepath" ]] && continue
|
||||||
|
|
||||||
|
if grep -qF "$filepath" "$TRACKED_FILE" 2>/dev/null; then
|
||||||
|
log "TRACKED: $filepath"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
if is_protected_file "$filepath"; then
|
||||||
|
log "PROTECTED: $filepath"
|
||||||
|
((PROTECTED_COUNT++))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
FILE_SIZE=$(stat -c%s "$filepath" 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
if is_video_file "$filepath"; then
|
||||||
|
FILE_MTIME=$(stat -c %Y "$filepath" 2>/dev/null || echo 0)
|
||||||
|
FILE_AGE=$(( NOW - FILE_MTIME ))
|
||||||
|
|
||||||
|
if [[ "$FILE_AGE" -lt "$AGE_SECONDS" ]]; then
|
||||||
|
log "RECENT (skipping): $filepath"
|
||||||
|
((RECENT_COUNT++))
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
warn "$ICON_TRASH ORPHAN: $filepath"
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
rm -f "$filepath" && {
|
||||||
|
((ORPHAN_COUNT++))
|
||||||
|
ORPHAN_BYTES=$((ORPHAN_BYTES + FILE_SIZE))
|
||||||
|
} || error "Failed to delete: $filepath"
|
||||||
|
else
|
||||||
|
((ORPHAN_COUNT++))
|
||||||
|
ORPHAN_BYTES=$((ORPHAN_BYTES + FILE_SIZE))
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
log "JUNK: $filepath"
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
rm -f "$filepath" && {
|
||||||
|
((JUNK_COUNT++))
|
||||||
|
JUNK_BYTES=$((JUNK_BYTES + FILE_SIZE))
|
||||||
|
} || error "Failed to delete: $filepath"
|
||||||
|
else
|
||||||
|
((JUNK_COUNT++))
|
||||||
|
JUNK_BYTES=$((JUNK_BYTES + FILE_SIZE))
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
done < <(find "$SONARR_TV_ROOT" -type f 2>/dev/null)
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
echo ""
|
||||||
|
info "Cleaning up empty folders..."
|
||||||
|
find "$SONARR_TV_ROOT" -mindepth 1 -type d -empty -delete 2>/dev/null
|
||||||
|
success "Empty folders removed"
|
||||||
|
fi
|
||||||
|
|
||||||
|
END=$(date +%s)
|
||||||
|
|
||||||
|
format_bytes() {
|
||||||
|
local bytes=$1
|
||||||
|
if (( bytes > 1073741824 )); then
|
||||||
|
awk "BEGIN {printf \"%.1fGB\", $bytes / 1073741824}"
|
||||||
|
elif (( bytes > 1048576 )); then
|
||||||
|
awk "BEGIN {printf \"%.1fMB\", $bytes / 1048576}"
|
||||||
|
else
|
||||||
|
echo "${bytes}B"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
ORPHAN_HUMAN=$(format_bytes $ORPHAN_BYTES)
|
||||||
|
JUNK_HUMAN=$(format_bytes $JUNK_BYTES)
|
||||||
|
TOTAL_REMOVED=$(( ORPHAN_COUNT + JUNK_COUNT ))
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY SONARR CLEANUP SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_SYNC Tracked by Sonarr: $TRACKED_COUNT files"
|
||||||
|
echo "$ICON_SHIELD Protected: $PROTECTED_COUNT files (artwork, subtitles, metadata)"
|
||||||
|
echo "$ICON_TRASH Orphans removed: $ORPHAN_COUNT files ($ORPHAN_HUMAN)"
|
||||||
|
echo "$ICON_TRASH Junk removed: $JUNK_COUNT files ($JUNK_HUMAN)"
|
||||||
|
echo "$ICON_TIME Recent skipped: $RECENT_COUNT files (under ${SONARR_ORPHAN_AGE} days)"
|
||||||
|
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||||
|
echo ""
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "$ICON_WARN Status: DRY RUN — no files deleted"
|
||||||
|
elif [[ "$TOTAL_REMOVED" -eq 0 ]]; then
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS CLEAN — nothing to remove"
|
||||||
|
notify "Sonarr cleanup complete on $(hostname) — library is clean" "Sonarr Cleanup" "normal"
|
||||||
|
else
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS DONE — $TOTAL_REMOVED files removed"
|
||||||
|
notify "Sonarr cleanup on $(hostname) — removed $TOTAL_REMOVED files (orphans: $ORPHAN_HUMAN junk: $JUNK_HUMAN)" "Sonarr Cleanup" "normal"
|
||||||
|
fi
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
@@ -0,0 +1,187 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- Backup Verify ----------------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Verifies the rsync mirror is healthy by comparing random file samples between
|
||||||
|
# local and remote servers using MD5 checksums.
|
||||||
|
#
|
||||||
|
# Randomly samples BACKUP_VERIFY_SAMPLE files per share, computes checksums locally,
|
||||||
|
# then computes the same checksums on the remote via SSH and compares results.
|
||||||
|
#
|
||||||
|
# Results per file:
|
||||||
|
# MATCH — checksums identical, file is correctly mirrored
|
||||||
|
# MISMATCH — file exists on both but checksums differ — sync may have failed
|
||||||
|
# MISSING — file exists locally but not on remote — not yet synced or deleted
|
||||||
|
#
|
||||||
|
# Silent when all files match. Notifies on any mismatch or missing file.
|
||||||
|
# Uses existing SSH keys — no additional configuration needed beyond share list.
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under Backup Verify section.
|
||||||
|
# Supports --dry-run to show what would be checked without running checksums.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_GEAR Setup ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||||
|
|
||||||
|
detect_hosts
|
||||||
|
resolve_remote_ip
|
||||||
|
|
||||||
|
# Use BACKUP_VERIFY_SHARES if defined, fall back to DAILY_SYNC_SHARES
|
||||||
|
if [[ ${#BACKUP_VERIFY_SHARES[@]} -gt 0 ]]; then
|
||||||
|
VERIFY_SHARES=("${BACKUP_VERIFY_SHARES[@]}")
|
||||||
|
info "Using BACKUP_VERIFY_SHARES (${#VERIFY_SHARES[@]} shares)"
|
||||||
|
else
|
||||||
|
VERIFY_SHARES=("${DAILY_SYNC_SHARES[@]}")
|
||||||
|
info "BACKUP_VERIFY_SHARES not set — using DAILY_SYNC_SHARES (${#VERIFY_SHARES[@]} shares)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ ${#VERIFY_SHARES[@]} -eq 0 ]]; then
|
||||||
|
warn "No shares configured — nothing to verify"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Status ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
if [[ "$SHOW_STATUS" == true ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||||
|
echo "$ICON_VERIFY Shares: ${#VERIFY_SHARES[@]}"
|
||||||
|
echo "$ICON_VERIFY Sample: $BACKUP_VERIFY_SAMPLE files per share"
|
||||||
|
echo "$ICON_VERIFY Min size: $BACKUP_VERIFY_MIN_SIZE"
|
||||||
|
echo "$ICON_HOST Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
|
||||||
|
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — showing sample selection only, no checksums computed"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_VERIFY Backup Verification ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_VERIFY Backup Verification — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||||
|
echo "$ICON_HOST Remote: $REMOTE_SERVER_NAME ($REMOTE_SERVER)"
|
||||||
|
echo "$ICON_VERIFY Sample: $BACKUP_VERIFY_SAMPLE files per share (min size: $BACKUP_VERIFY_MIN_SIZE)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
START=$(date +%s)
|
||||||
|
TOTAL_CHECKED=0
|
||||||
|
TOTAL_MATCH=0
|
||||||
|
TOTAL_MISMATCH=0
|
||||||
|
TOTAL_MISSING=0
|
||||||
|
SHARES_WITH_ISSUES=()
|
||||||
|
|
||||||
|
for share in "${VERIFY_SHARES[@]}"; do
|
||||||
|
SHARE_NAME=$(basename "$share")
|
||||||
|
echo "━━━ $ICON_VERIFY $SHARE_NAME ━━━"
|
||||||
|
|
||||||
|
if [[ ! -d "$share" ]]; then
|
||||||
|
warn "$SHARE_NAME not found locally — skipping"
|
||||||
|
echo ""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Find files above minimum size and randomly sample
|
||||||
|
SAMPLE_FILES=$(find "$share" -type f -size +"$BACKUP_VERIFY_MIN_SIZE" 2>/dev/null | \
|
||||||
|
shuf | head -n "$BACKUP_VERIFY_SAMPLE")
|
||||||
|
|
||||||
|
SAMPLE_COUNT=$(echo "$SAMPLE_FILES" | grep -c "." 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
if [[ "$SAMPLE_COUNT" -eq 0 ]]; then
|
||||||
|
info "No files found above $BACKUP_VERIFY_MIN_SIZE — skipping"
|
||||||
|
echo ""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
info "Sampled $SAMPLE_COUNT files"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "$SAMPLE_FILES" | while IFS= read -r f; do
|
||||||
|
warn "DRY RUN — would check: $(basename "$f")"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
SHARE_MISMATCH=0
|
||||||
|
SHARE_MISSING=0
|
||||||
|
SHARE_MATCH=0
|
||||||
|
|
||||||
|
while IFS= read -r local_file; do
|
||||||
|
[[ -z "$local_file" ]] && continue
|
||||||
|
|
||||||
|
# Compute local checksum
|
||||||
|
local_md5=$(md5sum "$local_file" 2>/dev/null | awk '{print $1}')
|
||||||
|
|
||||||
|
if [[ -z "$local_md5" ]]; then
|
||||||
|
warn "Could not checksum: $local_file — skipping"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Compute remote checksum via SSH
|
||||||
|
remote_md5=$(ssh -i "$SSH_KEY" -o ConnectTimeout=10 root@"$REMOTE_SERVER" \
|
||||||
|
"md5sum '$local_file' 2>/dev/null | awk '{print \$1}'" 2>/dev/null)
|
||||||
|
|
||||||
|
((TOTAL_CHECKED++))
|
||||||
|
|
||||||
|
if [[ -z "$remote_md5" ]]; then
|
||||||
|
warn "$ICON_ERROR MISSING: $(basename "$local_file")"
|
||||||
|
((SHARE_MISSING++))
|
||||||
|
((TOTAL_MISSING++))
|
||||||
|
elif [[ "$local_md5" == "$remote_md5" ]]; then
|
||||||
|
log "MATCH: $(basename "$local_file")"
|
||||||
|
((SHARE_MATCH++))
|
||||||
|
((TOTAL_MATCH++))
|
||||||
|
else
|
||||||
|
error "$ICON_ERROR MISMATCH: $(basename "$local_file")"
|
||||||
|
((SHARE_MISMATCH++))
|
||||||
|
((TOTAL_MISMATCH++))
|
||||||
|
fi
|
||||||
|
|
||||||
|
done <<< "$SAMPLE_FILES"
|
||||||
|
|
||||||
|
echo " $ICON_SUCCESS Match: $SHARE_MATCH $ICON_WARN Missing: $SHARE_MISSING $ICON_ERROR Mismatch: $SHARE_MISMATCH"
|
||||||
|
|
||||||
|
if [[ "$SHARE_MISMATCH" -gt 0 || "$SHARE_MISSING" -gt 0 ]]; then
|
||||||
|
SHARES_WITH_ISSUES+=("$SHARE_NAME")
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
|
||||||
|
END=$(date +%s)
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo "━━━━━ $ICON_SUMMARY BACKUP VERIFY SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_HOST Remote: $REMOTE_SERVER_NAME"
|
||||||
|
echo "$ICON_VERIFY Checked: $TOTAL_CHECKED files"
|
||||||
|
echo "$ICON_SUCCESS Match: $TOTAL_MATCH"
|
||||||
|
echo "$ICON_WARN Missing: $TOTAL_MISSING"
|
||||||
|
echo "$ICON_ERROR Mismatch: $TOTAL_MISMATCH"
|
||||||
|
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "$ICON_WARN Status: DRY RUN — no checksums computed"
|
||||||
|
elif [[ "$TOTAL_MISMATCH" -gt 0 || "$TOTAL_MISSING" -gt 0 ]]; then
|
||||||
|
echo "$ICON_ERROR Status: ISSUES FOUND — ${#SHARES_WITH_ISSUES[@]} share(s) need attention"
|
||||||
|
notify "Backup verify failed on $(hostname) → $REMOTE_SERVER_NAME — mismatches: $TOTAL_MISMATCH missing: $TOTAL_MISSING — shares: ${SHARES_WITH_ISSUES[*]}" "Backup Verify" "warning"
|
||||||
|
else
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS ALL FILES MATCH"
|
||||||
|
notify "Backup verify passed on $(hostname) → $REMOTE_SERVER_NAME — $TOTAL_CHECKED files checked across ${#VERIFY_SHARES[@]} shares" "Backup Verify" "normal"
|
||||||
|
fi
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
@@ -0,0 +1,152 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- Bandwidth Monitor ------------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Logs daily rsync transfer totals and generates weekly summary reports.
|
||||||
|
# Designed for minimal flash drive impact — one bounded write per day.
|
||||||
|
#
|
||||||
|
# Two modes:
|
||||||
|
# --log-transfer "profile" bytes duration — called by rsync.sh after each sync
|
||||||
|
# appends one line, trims old entries
|
||||||
|
# --report — generates weekly summary from log
|
||||||
|
# (no args) — generates summary report
|
||||||
|
#
|
||||||
|
# Log format (one line per transfer):
|
||||||
|
# YYYY-MM-DD|profile|bytes|duration_seconds
|
||||||
|
#
|
||||||
|
# Log file stays bounded to BANDWIDTH_LOG_RETENTION days — old entries trimmed on write.
|
||||||
|
# Minimal writes: one append per rsync run, one trim per append.
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under Bandwidth Monitor section.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# Check for log-transfer mode
|
||||||
|
LOG_TRANSFER_MODE=false
|
||||||
|
REPORT_MODE=false
|
||||||
|
TRANSFER_PROFILE=""
|
||||||
|
TRANSFER_BYTES=0
|
||||||
|
TRANSFER_DURATION=0
|
||||||
|
|
||||||
|
for arg in "${PARSED_ARGS[@]}"; do
|
||||||
|
case "$arg" in
|
||||||
|
--log-transfer) LOG_TRANSFER_MODE=true ;;
|
||||||
|
--report) REPORT_MODE=true ;;
|
||||||
|
*)
|
||||||
|
if [[ "$LOG_TRANSFER_MODE" == true && -z "$TRANSFER_PROFILE" ]]; then
|
||||||
|
TRANSFER_PROFILE="$arg"
|
||||||
|
elif [[ "$LOG_TRANSFER_MODE" == true && "$TRANSFER_BYTES" -eq 0 ]]; then
|
||||||
|
TRANSFER_BYTES="$arg"
|
||||||
|
elif [[ "$LOG_TRANSFER_MODE" == true ]]; then
|
||||||
|
TRANSFER_DURATION="$arg"
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# Ensure log file exists
|
||||||
|
touch "$BANDWIDTH_LOG" 2>/dev/null || {
|
||||||
|
error "Cannot create bandwidth log: $BANDWIDTH_LOG"
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# LOG TRANSFER MODE
|
||||||
|
# Called by rsync.sh after each successful sync — appends one line and trims old entries
|
||||||
|
# Usage: bandwidth_monitor.sh --log-transfer "profile" bytes duration
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
if [[ "$LOG_TRANSFER_MODE" == true ]]; then
|
||||||
|
TODAY=$(date '+%Y-%m-%d')
|
||||||
|
echo "${TODAY}|${TRANSFER_PROFILE}|${TRANSFER_BYTES}|${TRANSFER_DURATION}" >> "$BANDWIDTH_LOG"
|
||||||
|
log "Logged transfer: $TRANSFER_PROFILE — $TRANSFER_BYTES bytes in ${TRANSFER_DURATION}s"
|
||||||
|
|
||||||
|
# Trim entries older than retention period — keeps file bounded
|
||||||
|
CUTOFF=$(date -d "${BANDWIDTH_LOG_RETENTION} days ago" '+%Y-%m-%d')
|
||||||
|
TEMP_FILE="${BANDWIDTH_LOG}.tmp"
|
||||||
|
awk -F'|' -v cutoff="$CUTOFF" '$1 >= cutoff' "$BANDWIDTH_LOG" > "$TEMP_FILE"
|
||||||
|
mv "$TEMP_FILE" "$BANDWIDTH_LOG"
|
||||||
|
log "Log trimmed — keeping entries from $CUTOFF onwards"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# REPORT MODE — generate summary from log
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_BANDWIDTH Bandwidth Report — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
if [[ ! -s "$BANDWIDTH_LOG" ]]; then
|
||||||
|
warn "No bandwidth data yet — log is empty"
|
||||||
|
warn "Data accumulates as rsync jobs run"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
START=$(date +%s)
|
||||||
|
|
||||||
|
# Calculate date range in log
|
||||||
|
OLDEST=$(awk -F'|' 'NR==1{print $1}' "$BANDWIDTH_LOG")
|
||||||
|
NEWEST=$(awk -F'|' 'END{print $1}' "$BANDWIDTH_LOG")
|
||||||
|
ENTRY_COUNT=$(wc -l < "$BANDWIDTH_LOG")
|
||||||
|
|
||||||
|
info "Log covers: $OLDEST → $NEWEST ($ENTRY_COUNT entries)"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Total bytes transferred
|
||||||
|
TOTAL_BYTES=$(awk -F'|' '{sum += $3} END {print sum+0}' "$BANDWIDTH_LOG")
|
||||||
|
TOTAL_GB=$(awk "BEGIN {printf \"%.2f\", $TOTAL_BYTES / 1073741824}")
|
||||||
|
|
||||||
|
# Per-profile breakdown
|
||||||
|
echo "━━━ $ICON_BANDWIDTH Per-Profile Totals ━━━"
|
||||||
|
awk -F'|' '{
|
||||||
|
bytes[$2] += $3
|
||||||
|
runs[$2]++
|
||||||
|
duration[$2] += $4
|
||||||
|
}
|
||||||
|
END {
|
||||||
|
for (profile in bytes) {
|
||||||
|
gb = bytes[profile] / 1073741824
|
||||||
|
printf " %-20s %6.2f GB (%d runs)\n", profile, gb, runs[profile]
|
||||||
|
}
|
||||||
|
}' "$BANDWIDTH_LOG" | sort -k3 -rn
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Daily totals for the last 7 days
|
||||||
|
echo "━━━ $ICON_BANDWIDTH Last 7 Days ━━━"
|
||||||
|
for i in 6 5 4 3 2 1 0; do
|
||||||
|
day=$(date -d "$i days ago" '+%Y-%m-%d')
|
||||||
|
day_bytes=$(awk -F'|' -v d="$day" '$1==d{sum+=$3} END{print sum+0}' "$BANDWIDTH_LOG")
|
||||||
|
day_gb=$(awk "BEGIN {printf \"%.2f\", $day_bytes / 1073741824}")
|
||||||
|
|
||||||
|
# Flag days that exceeded warning threshold
|
||||||
|
over_warn=$(awk "BEGIN {print ($day_bytes > $BANDWIDTH_WARN_GB * 1073741824) ? 1 : 0}")
|
||||||
|
if [[ "$over_warn" == "1" ]]; then
|
||||||
|
echo " $ICON_WARN $day ${day_gb} GB ← exceeded ${BANDWIDTH_WARN_GB}GB threshold"
|
||||||
|
else
|
||||||
|
echo " $ICON_TIME $day ${day_gb} GB"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_SUMMARY Totals ━━━"
|
||||||
|
echo " $ICON_BANDWIDTH Total transferred: ${TOTAL_GB} GB"
|
||||||
|
echo " $ICON_TIME Log period: $OLDEST → $NEWEST"
|
||||||
|
echo " $ICON_GEAR Retention: ${BANDWIDTH_LOG_RETENTION} days"
|
||||||
|
|
||||||
|
END=$(date +%s)
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY BANDWIDTH SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_BANDWIDTH Total: ${TOTAL_GB} GB"
|
||||||
|
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
|
||||||
|
notify "Bandwidth report on $(hostname) — ${TOTAL_GB}GB transferred (${OLDEST} to ${NEWEST})" "Bandwidth Monitor" "normal"
|
||||||
@@ -0,0 +1,182 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- Certificate Monitor ----------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Monitors SSL certificate expiry for all configured domains by connecting directly
|
||||||
|
# via openssl — no dependency on NPM or any other service. Reads the actual certificate
|
||||||
|
# the server is presenting to the outside world.
|
||||||
|
#
|
||||||
|
# This approach catches real-world cert issues that API-based checks miss:
|
||||||
|
# - Cert renewed but server not reloaded
|
||||||
|
# - Wrong cert being served
|
||||||
|
# - Cert chain issues
|
||||||
|
#
|
||||||
|
# Each domain and subdomain is a separate entry — they have independent certs.
|
||||||
|
# Silent when all certs are healthy. Notifies when any approach warning threshold.
|
||||||
|
# Notifications batched per severity — one message for warnings, one for criticals.
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under Certificate Monitor section.
|
||||||
|
# Supports --dry-run to check certs and show results without sending notifications.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_GEAR Setup ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||||
|
|
||||||
|
if ! command -v openssl >/dev/null 2>&1; then
|
||||||
|
error "openssl not found — required for certificate checks"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "openssl available"
|
||||||
|
|
||||||
|
if [[ ${#CERT_MONITOR_DOMAINS[@]} -eq 0 ]]; then
|
||||||
|
warn "CERT_MONITOR_DOMAINS is empty in Master.conf — add your domains to enable monitoring"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
info "$ICON_CERT Domains to check: ${#CERT_MONITOR_DOMAINS[@]}"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Status ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
if [[ "$SHOW_STATUS" == true ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||||
|
echo "$ICON_CERT Domains: ${CERT_MONITOR_DOMAINS[*]}"
|
||||||
|
echo "$ICON_WARN Warn at: ${CERT_WARN_DAYS} days remaining"
|
||||||
|
echo "$ICON_ERROR Crit at: ${CERT_CRIT_DAYS} days remaining"
|
||||||
|
echo "$ICON_TIME Timeout: ${CERT_TIMEOUT}s per domain"
|
||||||
|
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — results shown but no notifications sent"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# CERT CHECK FUNCTION
|
||||||
|
# Connects to domain:443 via openssl, extracts expiry date, calculates days remaining.
|
||||||
|
# Returns 0=healthy 1=warning 2=critical 3=failed
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
check_cert() {
|
||||||
|
local domain="$1"
|
||||||
|
local port="${2:-443}"
|
||||||
|
|
||||||
|
local expiry_str
|
||||||
|
expiry_str=$(echo | timeout "$CERT_TIMEOUT" openssl s_client \
|
||||||
|
-connect "${domain}:${port}" \
|
||||||
|
-servername "$domain" \
|
||||||
|
2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
|
||||||
|
|
||||||
|
if [[ -z "$expiry_str" ]]; then
|
||||||
|
error "$ICON_CERT $domain — could not retrieve certificate"
|
||||||
|
return 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
local expiry_epoch
|
||||||
|
expiry_epoch=$(date -d "$expiry_str" +%s 2>/dev/null)
|
||||||
|
|
||||||
|
if [[ -z "$expiry_epoch" ]]; then
|
||||||
|
error "$ICON_CERT $domain — could not parse expiry date: $expiry_str"
|
||||||
|
return 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
local now days_remaining expiry_display
|
||||||
|
now=$(date +%s)
|
||||||
|
days_remaining=$(( (expiry_epoch - now) / 86400 ))
|
||||||
|
expiry_display=$(date -d "$expiry_str" '+%Y-%m-%d' 2>/dev/null)
|
||||||
|
|
||||||
|
if [[ "$days_remaining" -le "$CERT_CRIT_DAYS" ]]; then
|
||||||
|
error "$ICON_CERT $domain — CRITICAL: ${days_remaining} days remaining (expires $expiry_display)"
|
||||||
|
return 2
|
||||||
|
elif [[ "$days_remaining" -le "$CERT_WARN_DAYS" ]]; then
|
||||||
|
warn "$ICON_CERT $domain — WARNING: ${days_remaining} days remaining (expires $expiry_display)"
|
||||||
|
return 1
|
||||||
|
else
|
||||||
|
success "$ICON_CERT $domain — OK: ${days_remaining} days remaining (expires $expiry_display)"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_CERT Certificate Monitor ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_CERT Certificate Monitor — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||||
|
echo "$ICON_WARN Warn threshold: ${CERT_WARN_DAYS} days"
|
||||||
|
echo "$ICON_ERROR Crit threshold: ${CERT_CRIT_DAYS} days"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
START=$(date +%s)
|
||||||
|
HEALTHY=()
|
||||||
|
WARNING=()
|
||||||
|
CRITICAL=()
|
||||||
|
FAILED=()
|
||||||
|
declare -A DOMAIN_STATUS
|
||||||
|
|
||||||
|
for domain in "${CERT_MONITOR_DOMAINS[@]}"; do
|
||||||
|
[[ -z "$domain" ]] && continue
|
||||||
|
echo "━━━ $ICON_CERT $domain ━━━"
|
||||||
|
check_cert "$domain"
|
||||||
|
result=$?
|
||||||
|
case $result in
|
||||||
|
0) HEALTHY+=("$domain"); DOMAIN_STATUS["$domain"]="OK" ;;
|
||||||
|
1) WARNING+=("$domain"); DOMAIN_STATUS["$domain"]="WARN" ;;
|
||||||
|
2) CRITICAL+=("$domain"); DOMAIN_STATUS["$domain"]="CRIT" ;;
|
||||||
|
3) FAILED+=("$domain"); DOMAIN_STATUS["$domain"]="FAIL" ;;
|
||||||
|
esac
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
|
||||||
|
END=$(date +%s)
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == false ]]; then
|
||||||
|
[[ ${#CRITICAL[@]} -gt 0 ]] && \
|
||||||
|
notify "Certificate CRITICAL on $(hostname) — expiring within ${CERT_CRIT_DAYS} days: ${CRITICAL[*]}" "Certificate Monitor" "warning"
|
||||||
|
[[ ${#WARNING[@]} -gt 0 ]] && \
|
||||||
|
notify "Certificate WARNING on $(hostname) — expiring within ${CERT_WARN_DAYS} days: ${WARNING[*]}" "Certificate Monitor" "warning"
|
||||||
|
[[ ${#FAILED[@]} -gt 0 ]] && \
|
||||||
|
notify "Certificate check FAILED on $(hostname) — could not reach: ${FAILED[*]}" "Certificate Monitor" "warning"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo "━━━━━ $ICON_SUMMARY CERTIFICATE MONITOR SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||||
|
echo ""
|
||||||
|
echo " $ICON_SUCCESS Healthy: ${#HEALTHY[@]} $ICON_WARN Warning: ${#WARNING[@]} $ICON_ERROR Critical: ${#CRITICAL[@]} Failed: ${#FAILED[@]}"
|
||||||
|
echo ""
|
||||||
|
for domain in "${CERT_MONITOR_DOMAINS[@]}"; do
|
||||||
|
[[ -z "$domain" ]] && continue
|
||||||
|
case "${DOMAIN_STATUS[$domain]:-UNKN}" in
|
||||||
|
OK) echo " $ICON_SUCCESS $domain" ;;
|
||||||
|
WARN) echo " $ICON_WARN $domain" ;;
|
||||||
|
CRIT) echo " $ICON_ERROR $domain" ;;
|
||||||
|
FAIL) echo " $ICON_ERROR $domain (unreachable)" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "$ICON_WARN Status: DRY RUN — no notifications sent"
|
||||||
|
elif [[ ${#CRITICAL[@]} -gt 0 || ${#FAILED[@]} -gt 0 ]]; then
|
||||||
|
echo "$ICON_ERROR Status: ACTION REQUIRED"
|
||||||
|
elif [[ ${#WARNING[@]} -gt 0 ]]; then
|
||||||
|
echo "$ICON_WARN Status: WARNINGS — renewal recommended"
|
||||||
|
else
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS ALL CERTS HEALTHY"
|
||||||
|
fi
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
|
||||||
|
[[ ${#CRITICAL[@]} -gt 0 || ${#FAILED[@]} -gt 0 ]] && exit 1
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,249 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- SMART Health Monitor ---------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Checks SMART health attributes for all drives on the system.
|
||||||
|
# Reads data live from each drive via smartctl — no persistent writes.
|
||||||
|
#
|
||||||
|
# Monitored attributes:
|
||||||
|
# Reallocated_Sector_Ct — bad sectors remapped — any > 0 is concerning
|
||||||
|
# Current_Pending_Sector — sectors waiting for reallocation — any > 0 is concerning
|
||||||
|
# Offline_Uncorrectable — sectors that could not be corrected — any > 0 is critical
|
||||||
|
# Temperature_Celsius — drive temperature vs SMART_TEMP_WARN / SMART_TEMP_CRIT
|
||||||
|
# Power_On_Hours — informational — drive age estimation
|
||||||
|
# SMART overall status — pass/fail per drive
|
||||||
|
#
|
||||||
|
# Discovers drives automatically — no configuration needed for drive list.
|
||||||
|
# SMART_IGNORE_DRIVES allows skipping specific drives (e.g. USB flash drives).
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under SMART Health section.
|
||||||
|
# Supports --dry-run to show which drives would be checked without running smartctl.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_GEAR Setup ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||||
|
|
||||||
|
if [[ "$EUID" -ne 0 ]]; then
|
||||||
|
error "Must be run as root"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "Running as root"
|
||||||
|
|
||||||
|
if ! command -v smartctl >/dev/null 2>&1; then
|
||||||
|
error "smartctl not found — install smartmontools"
|
||||||
|
notify "SMART health check failed on $(hostname) — smartmontools not installed" "SMART Health" "warning"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
success "smartctl available"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Status ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
if [[ "$SHOW_STATUS" == true ]]; then
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||||
|
echo "$ICON_SMART Temp warn: ${SMART_TEMP_WARN}°C"
|
||||||
|
echo "$ICON_SMART Temp crit: ${SMART_TEMP_CRIT}°C"
|
||||||
|
echo "$ICON_SMART Ignore drives: ${SMART_IGNORE_DRIVES[*]:-none}"
|
||||||
|
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Show drives that would be checked
|
||||||
|
echo "━━━ Discovered Drives ━━━"
|
||||||
|
for drive in /dev/sd? /dev/nvme?; do
|
||||||
|
[[ ! -e "$drive" ]] && continue
|
||||||
|
drive_name=$(basename "$drive")
|
||||||
|
ignored=false
|
||||||
|
for ignore in "${SMART_IGNORE_DRIVES[@]}"; do
|
||||||
|
[[ "$drive_name" == "$ignore" ]] && ignored=true && break
|
||||||
|
done
|
||||||
|
if [[ "$ignored" == true ]]; then
|
||||||
|
echo " $ICON_WARN $drive — ignored"
|
||||||
|
else
|
||||||
|
echo " $ICON_SMART $drive — would check"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — showing drive list only, no SMART data read"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# HELPER — extract SMART attribute value
|
||||||
|
# Usage: get_smart_attr "/dev/sda" "Reallocated_Sector_Ct"
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
get_smart_attr() {
|
||||||
|
local drive="$1" attr="$2"
|
||||||
|
smartctl -A "$drive" 2>/dev/null | \
|
||||||
|
awk -v attr="$attr" '$2 == attr {print $10}'
|
||||||
|
}
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SMART SMART Health Check ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_SMART SMART Health Check — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
START=$(date +%s)
|
||||||
|
DRIVES_OK=()
|
||||||
|
DRIVES_WARN=()
|
||||||
|
DRIVES_CRIT=()
|
||||||
|
DRIVES_SKIP=()
|
||||||
|
|
||||||
|
for drive in /dev/sd? /dev/nvme?; do
|
||||||
|
[[ ! -e "$drive" ]] && continue
|
||||||
|
drive_name=$(basename "$drive")
|
||||||
|
|
||||||
|
# Check ignore list
|
||||||
|
ignored=false
|
||||||
|
for ignore in "${SMART_IGNORE_DRIVES[@]}"; do
|
||||||
|
[[ "$drive_name" == "$ignore" ]] && ignored=true && break
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ "$ignored" == true ]]; then
|
||||||
|
info "$drive_name — ignored (in SMART_IGNORE_DRIVES)"
|
||||||
|
DRIVES_SKIP+=("$drive_name")
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "━━━ $ICON_SMART $drive_name ━━━"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
warn "DRY RUN — would check $drive_name"
|
||||||
|
echo ""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if drive supports SMART
|
||||||
|
if ! smartctl -i "$drive" 2>/dev/null | grep -q "SMART support is: Enabled"; then
|
||||||
|
warn "$drive_name — SMART not enabled or not supported"
|
||||||
|
DRIVES_SKIP+=("$drive_name")
|
||||||
|
echo ""
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Overall SMART status
|
||||||
|
SMART_STATUS=$(smartctl -H "$drive" 2>/dev/null | grep "overall-health" | awk '{print $NF}')
|
||||||
|
if [[ "$SMART_STATUS" == "PASSED" ]]; then
|
||||||
|
success "Overall status: PASSED"
|
||||||
|
else
|
||||||
|
error "Overall status: $SMART_STATUS"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Key attributes
|
||||||
|
DRIVE_WARN=false
|
||||||
|
DRIVE_CRIT=false
|
||||||
|
|
||||||
|
# Reallocated sectors
|
||||||
|
REALLOC=$(get_smart_attr "$drive" "Reallocated_Sector_Ct")
|
||||||
|
if [[ -n "$REALLOC" ]]; then
|
||||||
|
if [[ "$REALLOC" -gt 0 ]]; then
|
||||||
|
warn "$ICON_SMART Reallocated sectors: $REALLOC — drive showing wear"
|
||||||
|
DRIVE_WARN=true
|
||||||
|
else
|
||||||
|
success "$ICON_SMART Reallocated sectors: $REALLOC"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Pending sectors
|
||||||
|
PENDING=$(get_smart_attr "$drive" "Current_Pending_Sector")
|
||||||
|
if [[ -n "$PENDING" ]]; then
|
||||||
|
if [[ "$PENDING" -gt 0 ]]; then
|
||||||
|
warn "$ICON_SMART Pending sectors: $PENDING — sectors awaiting reallocation"
|
||||||
|
DRIVE_WARN=true
|
||||||
|
else
|
||||||
|
success "$ICON_SMART Pending sectors: $PENDING"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Uncorrectable sectors
|
||||||
|
UNCORR=$(get_smart_attr "$drive" "Offline_Uncorrectable")
|
||||||
|
if [[ -n "$UNCORR" ]]; then
|
||||||
|
if [[ "$UNCORR" -gt 0 ]]; then
|
||||||
|
error "$ICON_SMART Uncorrectable sectors: $UNCORR — CRITICAL"
|
||||||
|
DRIVE_CRIT=true
|
||||||
|
else
|
||||||
|
success "$ICON_SMART Uncorrectable sectors: $UNCORR"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Temperature
|
||||||
|
TEMP=$(get_smart_attr "$drive" "Temperature_Celsius")
|
||||||
|
# NVMe uses different attribute name
|
||||||
|
[[ -z "$TEMP" ]] && TEMP=$(smartctl -A "$drive" 2>/dev/null | \
|
||||||
|
awk '/Temperature/{print $2}' | head -1)
|
||||||
|
|
||||||
|
if [[ -n "$TEMP" ]]; then
|
||||||
|
if [[ "$TEMP" -ge "$SMART_TEMP_CRIT" ]]; then
|
||||||
|
error "$ICON_SMART Temperature: ${TEMP}°C — CRITICAL (threshold: ${SMART_TEMP_CRIT}°C)"
|
||||||
|
DRIVE_CRIT=true
|
||||||
|
elif [[ "$TEMP" -ge "$SMART_TEMP_WARN" ]]; then
|
||||||
|
warn "$ICON_SMART Temperature: ${TEMP}°C — warning (threshold: ${SMART_TEMP_WARN}°C)"
|
||||||
|
DRIVE_WARN=true
|
||||||
|
else
|
||||||
|
success "$ICON_SMART Temperature: ${TEMP}°C"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Power on hours — informational
|
||||||
|
POH=$(get_smart_attr "$drive" "Power_On_Hours")
|
||||||
|
if [[ -n "$POH" ]]; then
|
||||||
|
POH_DAYS=$(( POH / 24 ))
|
||||||
|
info "$ICON_SMART Power on hours: $POH (${POH_DAYS} days)"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Classify drive
|
||||||
|
if [[ "$DRIVE_CRIT" == true ]]; then
|
||||||
|
DRIVES_CRIT+=("$drive_name")
|
||||||
|
elif [[ "$DRIVE_WARN" == true ]]; then
|
||||||
|
DRIVES_WARN+=("$drive_name")
|
||||||
|
else
|
||||||
|
DRIVES_OK+=("$drive_name")
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
done
|
||||||
|
|
||||||
|
END=$(date +%s)
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo "━━━━━ $ICON_SUMMARY SMART HEALTH SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||||
|
echo ""
|
||||||
|
echo " $ICON_SUCCESS Healthy: ${#DRIVES_OK[@]} $ICON_WARN Warning: ${#DRIVES_WARN[@]} $ICON_ERROR Critical: ${#DRIVES_CRIT[@]} skipped: ${#DRIVES_SKIP[@]}"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
[[ ${#DRIVES_OK[@]} -gt 0 ]] && echo " $ICON_SUCCESS ${DRIVES_OK[*]}"
|
||||||
|
[[ ${#DRIVES_WARN[@]} -gt 0 ]] && echo " $ICON_WARN ${DRIVES_WARN[*]}"
|
||||||
|
[[ ${#DRIVES_CRIT[@]} -gt 0 ]] && echo " $ICON_ERROR ${DRIVES_CRIT[*]}"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
echo "$ICON_WARN Status: DRY RUN"
|
||||||
|
elif [[ ${#DRIVES_CRIT[@]} -gt 0 ]]; then
|
||||||
|
echo "$ICON_ERROR Status: CRITICAL — ${DRIVES_CRIT[*]}"
|
||||||
|
notify "SMART CRITICAL on $(hostname) — drives need immediate attention: ${DRIVES_CRIT[*]}" "SMART Health" "warning"
|
||||||
|
elif [[ ${#DRIVES_WARN[@]} -gt 0 ]]; then
|
||||||
|
echo "$ICON_WARN Status: WARNING — ${DRIVES_WARN[*]}"
|
||||||
|
notify "SMART WARNING on $(hostname) — drives showing wear: ${DRIVES_WARN[*]}" "SMART Health" "warning"
|
||||||
|
else
|
||||||
|
echo "$ICON_DONE Status: $ICON_SUCCESS ALL DRIVES HEALTHY"
|
||||||
|
notify "SMART health check passed on $(hostname) — ${#DRIVES_OK[@]} drives healthy" "SMART Health" "normal"
|
||||||
|
fi
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
@@ -0,0 +1,245 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# --------------------------------- Health Digest ----------------------------------------------
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Aggregates system health data from across the ecosystem into a single digest report.
|
||||||
|
# Reads existing state files — no new writes to flash drive.
|
||||||
|
#
|
||||||
|
# Three profiles controlled by DIGEST_PROFILE in Master.conf:
|
||||||
|
# always — sends every run regardless of findings (schedule daily for daily digest)
|
||||||
|
# smart — sends only if something worth reporting was found (intelligent filtering)
|
||||||
|
# weekly — sends once per week on DIGEST_DAY regardless of schedule frequency
|
||||||
|
#
|
||||||
|
# The cron schedule stays the same regardless of profile — just change DIGEST_PROFILE
|
||||||
|
# in Master.conf to switch behavior. Run daily, profile controls when it actually notifies.
|
||||||
|
#
|
||||||
|
# Data sources (reads only — no writes):
|
||||||
|
# /tmp/transcode_state.db — ramdisk symlink and usage
|
||||||
|
# /tmp/container_watchdog_state.db — active container strikes
|
||||||
|
# /tmp/system_watchdog_state.db — active system strikes
|
||||||
|
# /boot/config/failover_state.db — current failover state
|
||||||
|
# /boot/config/system_watchdog_failed.db — container skip list
|
||||||
|
# /boot/config/bandwidth_history.db — recent transfer totals
|
||||||
|
# SSL certs via openssl (live check) — days remaining per domain
|
||||||
|
#
|
||||||
|
# All configuration in Master.conf under Health Digest section.
|
||||||
|
# Supports --dry-run to generate report without sending notification.
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
source "$SCRIPT_DIR/../Master.conf"
|
||||||
|
source "$SCRIPT_DIR/../common.sh"
|
||||||
|
|
||||||
|
parse_args "$@"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_GEAR Setup ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||||
|
|
||||||
|
success "Health Digest — profile: $DIGEST_PROFILE"
|
||||||
|
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — report generated but no notification sent"
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# Profile check — should we send today?
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
SHOULD_SEND=false
|
||||||
|
|
||||||
|
case "$DIGEST_PROFILE" in
|
||||||
|
always)
|
||||||
|
SHOULD_SEND=true
|
||||||
|
log "Profile: always — will send"
|
||||||
|
;;
|
||||||
|
weekly)
|
||||||
|
TODAY_NAME=$(date '+%A')
|
||||||
|
if [[ "$TODAY_NAME" == "$DIGEST_DAY" ]]; then
|
||||||
|
SHOULD_SEND=true
|
||||||
|
log "Profile: weekly — today is $DIGEST_DAY, will send"
|
||||||
|
else
|
||||||
|
info "Profile: weekly — today is $TODAY_NAME, digest day is $DIGEST_DAY — skipping"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
smart)
|
||||||
|
log "Profile: smart — will evaluate findings before deciding"
|
||||||
|
SHOULD_SEND=false # determined after gathering data
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
warn "Unknown DIGEST_PROFILE: $DIGEST_PROFILE — defaulting to weekly behavior"
|
||||||
|
TODAY_NAME=$(date '+%A')
|
||||||
|
[[ "$TODAY_NAME" == "$DIGEST_DAY" ]] && SHOULD_SEND=true
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# DATA GATHERING
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_DIGEST Gathering System Data ━━━"
|
||||||
|
|
||||||
|
FINDINGS=() # things worth noting
|
||||||
|
ISSUES=() # things that need attention
|
||||||
|
DIGEST_LINES=() # full report lines
|
||||||
|
|
||||||
|
# ── Failover State ──────────────────────────────────────────────────────────────────────────
|
||||||
|
if [[ -f "$FAILOVER_STATE_FILE" ]]; then
|
||||||
|
FAILOVER_STATE=$(grep "^state=" "$FAILOVER_STATE_FILE" 2>/dev/null | cut -d= -f2)
|
||||||
|
FAILOVER_CHANGE=$(grep "^last_change=" "$FAILOVER_STATE_FILE" 2>/dev/null | cut -d= -f2)
|
||||||
|
DIGEST_LINES+=("$ICON_FAILOVER Failover: $FAILOVER_STATE (last change: ${FAILOVER_CHANGE:-unknown})")
|
||||||
|
|
||||||
|
if [[ "$FAILOVER_STATE" != "NORMAL" && -n "$FAILOVER_STATE" ]]; then
|
||||||
|
ISSUES+=("Failover state: $FAILOVER_STATE")
|
||||||
|
[[ "$DIGEST_SMART_ON_FAILOVER" == true ]] && SHOULD_SEND=true
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_FAILOVER Failover: state file not found")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Transcode Ramdisk ───────────────────────────────────────────────────────────────────────
|
||||||
|
if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
|
||||||
|
RAMDISK_USED_KB=$(df "$RAMDISK_PATH" --output=used | tail -1 | tr -d ' ')
|
||||||
|
RAMDISK_USED_GB=$(awk "BEGIN {printf \"%.2f\", $RAMDISK_USED_KB / 1048576}")
|
||||||
|
RAMDISK_AVAIL_KB=$(df "$RAMDISK_PATH" --output=avail | tail -1 | tr -d ' ')
|
||||||
|
RAMDISK_AVAIL_GB=$(awk "BEGIN {printf \"%.2f\", $RAMDISK_AVAIL_KB / 1048576}")
|
||||||
|
SYMLINK_TARGET=$(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "unknown")
|
||||||
|
DIGEST_LINES+=("$ICON_RAM Transcodes: ${RAMDISK_USED_GB}GB used / ${RAMDISK_AVAIL_GB}GB free → $SYMLINK_TARGET")
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_RAM Transcodes: ramdisk not mounted")
|
||||||
|
ISSUES+=("Ramdisk not mounted")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Container Watchdog Strikes ──────────────────────────────────────────────────────────────
|
||||||
|
if [[ -f "$WATCHDOG_STATE_FILE" ]]; then
|
||||||
|
ACTIVE_STRIKES=$(grep -v ":0$" "$WATCHDOG_STATE_FILE" 2>/dev/null | wc -l)
|
||||||
|
if [[ "$ACTIVE_STRIKES" -gt 0 ]]; then
|
||||||
|
STRIKE_LIST=$(grep -v ":0$" "$WATCHDOG_STATE_FILE" 2>/dev/null | tr '\n' ' ')
|
||||||
|
DIGEST_LINES+=("$ICON_WATCHDOG Container strikes: $ACTIVE_STRIKES active — $STRIKE_LIST")
|
||||||
|
FINDINGS+=("Container watchdog: $ACTIVE_STRIKES active strikes")
|
||||||
|
[[ "$DIGEST_SMART_ON_WATCHDOG" == true ]] && SHOULD_SEND=true
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_WATCHDOG Container watchdog: no active strikes")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── System Watchdog Strikes ─────────────────────────────────────────────────────────────────
|
||||||
|
if [[ -f "$SYS_WATCHDOG_STATE_FILE" ]]; then
|
||||||
|
SYS_STRIKES=$(grep -v ":0$" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | wc -l)
|
||||||
|
if [[ "$SYS_STRIKES" -gt 0 ]]; then
|
||||||
|
SYS_STRIKE_LIST=$(grep -v ":0$" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | tr '\n' ' ')
|
||||||
|
DIGEST_LINES+=("$ICON_REBOOT_SMART System strikes: $SYS_STRIKES active — $SYS_STRIKE_LIST")
|
||||||
|
FINDINGS+=("System watchdog: $SYS_STRIKES active strikes")
|
||||||
|
[[ "$DIGEST_SMART_ON_WATCHDOG" == true ]] && SHOULD_SEND=true
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_REBOOT_SMART System watchdog: no active strikes")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Container Skip List ─────────────────────────────────────────────────────────────────────
|
||||||
|
if [[ -f "$SYS_WATCHDOG_FAILED_FILE" ]] && [[ -s "$SYS_WATCHDOG_FAILED_FILE" ]]; then
|
||||||
|
SKIP_COUNT=$(wc -l < "$SYS_WATCHDOG_FAILED_FILE")
|
||||||
|
SKIP_LIST=$(cat "$SYS_WATCHDOG_FAILED_FILE" | tr '\n' ' ')
|
||||||
|
DIGEST_LINES+=("$ICON_NOT_RUNNING Skip list: $SKIP_COUNT containers — $SKIP_LIST")
|
||||||
|
ISSUES+=("Containers on skip list: $SKIP_LIST")
|
||||||
|
SHOULD_SEND=true
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_RUNNING Skip list: empty — all containers healthy")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Bandwidth (yesterday's total) ───────────────────────────────────────────────────────────
|
||||||
|
if [[ -f "$BANDWIDTH_LOG" ]] && [[ -s "$BANDWIDTH_LOG" ]]; then
|
||||||
|
YESTERDAY=$(date -d "yesterday" '+%Y-%m-%d')
|
||||||
|
YESTERDAY_BYTES=$(awk -F'|' -v d="$YESTERDAY" '$1==d{sum+=$3} END{print sum+0}' "$BANDWIDTH_LOG")
|
||||||
|
YESTERDAY_GB=$(awk "BEGIN {printf \"%.2f\", $YESTERDAY_BYTES / 1073741824}")
|
||||||
|
OVER_WARN=$(awk "BEGIN {print ($YESTERDAY_BYTES > $BANDWIDTH_WARN_GB * 1073741824) ? 1 : 0}")
|
||||||
|
|
||||||
|
if [[ "$OVER_WARN" == "1" ]]; then
|
||||||
|
DIGEST_LINES+=("$ICON_BANDWIDTH Yesterday's transfers: ${YESTERDAY_GB}GB ← exceeded ${BANDWIDTH_WARN_GB}GB threshold")
|
||||||
|
FINDINGS+=("High bandwidth day: ${YESTERDAY_GB}GB transferred")
|
||||||
|
[[ "$DIGEST_SMART_ON_BANDWIDTH" == true ]] && SHOULD_SEND=true
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_BANDWIDTH Yesterday's transfers: ${YESTERDAY_GB}GB")
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_BANDWIDTH Bandwidth: no data yet")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── SSL Certificates ────────────────────────────────────────────────────────────────────────
|
||||||
|
if [[ ${#CERT_MONITOR_DOMAINS[@]} -gt 0 ]] && command -v openssl >/dev/null 2>&1; then
|
||||||
|
CERT_ISSUES=()
|
||||||
|
for domain in "${CERT_MONITOR_DOMAINS[@]}"; do
|
||||||
|
[[ -z "$domain" ]] && continue
|
||||||
|
expiry_str=$(echo | timeout "$CERT_TIMEOUT" openssl s_client \
|
||||||
|
-connect "${domain}:443" -servername "$domain" \
|
||||||
|
2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
|
||||||
|
|
||||||
|
if [[ -n "$expiry_str" ]]; then
|
||||||
|
expiry_epoch=$(date -d "$expiry_str" +%s 2>/dev/null)
|
||||||
|
days_remaining=$(( (expiry_epoch - $(date +%s)) / 86400 ))
|
||||||
|
|
||||||
|
if [[ "$days_remaining" -le "$CERT_CRIT_DAYS" ]]; then
|
||||||
|
CERT_ISSUES+=("$domain: ${days_remaining}d CRITICAL")
|
||||||
|
SHOULD_SEND=true
|
||||||
|
elif [[ "$days_remaining" -le "$CERT_WARN_DAYS" ]]; then
|
||||||
|
CERT_ISSUES+=("$domain: ${days_remaining}d WARNING")
|
||||||
|
[[ "$DIGEST_SMART_ON_CERT_WARN" == true ]] && SHOULD_SEND=true
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ ${#CERT_ISSUES[@]} -gt 0 ]]; then
|
||||||
|
DIGEST_LINES+=("$ICON_CERT Certificates: ${CERT_ISSUES[*]}")
|
||||||
|
FINDINGS+=("Cert issues: ${CERT_ISSUES[*]}")
|
||||||
|
else
|
||||||
|
DIGEST_LINES+=("$ICON_CERT Certificates: all healthy")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Smart profile final decision ────────────────────────────────────────────────────────────
|
||||||
|
if [[ "$DIGEST_PROFILE" == "smart" && "$SHOULD_SEND" == false ]]; then
|
||||||
|
info "Profile: smart — no findings worth reporting — skipping notification"
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY DIGEST SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_SUCCESS Everything looks healthy — no digest sent (smart profile)"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
# ━━━ $ICON_DIGEST Build and Send Digest ━━━
|
||||||
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
echo ""
|
||||||
|
echo "━━━ $ICON_DIGEST Health Digest ━━━"
|
||||||
|
|
||||||
|
DIGEST_DATE=$(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
DIGEST_HOST=$(hostname)
|
||||||
|
|
||||||
|
# Build notification message
|
||||||
|
NOTIFY_MSG="Health Digest — $DIGEST_HOST — $DIGEST_DATE"
|
||||||
|
if [[ ${#ISSUES[@]} -gt 0 ]]; then
|
||||||
|
NOTIFY_MSG+=" | Issues: ${ISSUES[*]}"
|
||||||
|
fi
|
||||||
|
if [[ ${#FINDINGS[@]} -gt 0 ]]; then
|
||||||
|
NOTIFY_MSG+=" | Findings: ${FINDINGS[*]}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Print full digest to console
|
||||||
|
for line in "${DIGEST_LINES[@]}"; do
|
||||||
|
echo " $line"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "━━━━━ $ICON_SUMMARY DIGEST SUMMARY ━━━━━"
|
||||||
|
echo "$ICON_DIGEST Profile: $DIGEST_PROFILE"
|
||||||
|
echo "$ICON_TIME Generated: $DIGEST_DATE"
|
||||||
|
echo "$ICON_ERROR Issues: ${#ISSUES[@]}"
|
||||||
|
echo "$ICON_WARN Findings: ${#FINDINGS[@]}"
|
||||||
|
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" == true ]]; then
|
||||||
|
warn "DRY RUN — digest generated but not sent"
|
||||||
|
elif [[ "$SHOULD_SEND" == true ]]; then
|
||||||
|
notify "$NOTIFY_MSG" "Health Digest" "$([[ ${#ISSUES[@]} -gt 0 ]] && echo "warning" || echo "normal")"
|
||||||
|
success "Digest sent"
|
||||||
|
fi
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# ----------------- UNRAID OPS COMMON LIBRARY (STABLE FRAMEWORK v1) ----------------------------
|
# ----------------- UNRAID OPS COMMON LIBRARY (STABLE FRAMEWORK v1) ----------------------------
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# Version: 2.7
|
# Version: 2.9
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# Changelog:
|
# Changelog:
|
||||||
# v1.0 — Initial stable framework
|
# v1.0 — Initial stable framework
|
||||||
@@ -51,11 +51,20 @@
|
|||||||
# ping_internet added — non-fatal external ping for failover use
|
# ping_internet added — non-fatal external ping for failover use
|
||||||
# v2.7 — ICON_WEBGUI added for WebGUI watchdog operations
|
# v2.7 — ICON_WEBGUI added for WebGUI watchdog operations
|
||||||
# ICON_DOCKER_NET added for Docker network connect operations
|
# ICON_DOCKER_NET added for Docker network connect operations
|
||||||
|
# v2.8 — ICON_CERT added for SSL certificate monitoring operations
|
||||||
|
# v2.9 — ICON_MONITOR added for monitoring section headers
|
||||||
|
# ICON_SMART added for drive SMART health operations
|
||||||
|
# ICON_BANDWIDTH added for bandwidth tracking operations
|
||||||
|
# ICON_DIGEST added for health digest operations
|
||||||
|
# ICON_EMBY added for Emby session reporting
|
||||||
|
# ICON_VERIFY added for backup verification operations
|
||||||
|
# Monitor/ folder added to ecosystem
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# ICONS
|
# ICONS
|
||||||
# Each icon has one job — do not reuse across different contexts.
|
# Each icon has one job — do not reuse across different contexts.
|
||||||
|
# Adding a new icon: add it to the appropriate group below with a comment describing its job.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
# System / Host
|
# System / Host
|
||||||
@@ -117,6 +126,17 @@ ICON_FAILOVER="🔀" # failover state changes and operations
|
|||||||
# Docker Network Operations
|
# Docker Network Operations
|
||||||
ICON_DOCKER_NET="🔌" # Docker network connect operations
|
ICON_DOCKER_NET="🔌" # Docker network connect operations
|
||||||
|
|
||||||
|
# Security / Certificate Operations
|
||||||
|
ICON_CERT="🔒" # SSL certificate monitoring
|
||||||
|
|
||||||
|
# Monitor Operations
|
||||||
|
ICON_MONITOR="📈" # monitoring section headers and general monitoring
|
||||||
|
ICON_SMART="🔧" # drive SMART health attribute monitoring
|
||||||
|
ICON_BANDWIDTH="📶" # bandwidth usage tracking and reporting
|
||||||
|
ICON_DIGEST="📰" # health digest — aggregated system summary
|
||||||
|
ICON_EMBY="🎬" # Emby media server session reporting
|
||||||
|
ICON_VERIFY="✔️" # backup verification — checksum comparison
|
||||||
|
|
||||||
# Notifications
|
# Notifications
|
||||||
ICON_NOTIFY="🔔" # notification operations
|
ICON_NOTIFY="🔔" # notification operations
|
||||||
|
|
||||||
@@ -128,6 +148,8 @@ ICON_SUCCESS="✅"
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# OUTPUT HELPERS
|
# OUTPUT HELPERS
|
||||||
|
# Standardised output functions used across all scripts.
|
||||||
|
# log() is gated by ENABLE_LOGGING — set in Master.conf or via --log flag.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
info() { echo "$ICON_INFO [INFO] $*"; }
|
info() { echo "$ICON_INFO [INFO] $*"; }
|
||||||
warn() { echo "$ICON_WARN [WARN] $*"; }
|
warn() { echo "$ICON_WARN [WARN] $*"; }
|
||||||
@@ -140,8 +162,10 @@ log() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# NOTIFICATION
|
# NOTIFICATION
|
||||||
|
# Sends a notification via unRAID native system and/or Discord webhook.
|
||||||
|
# Both channels are optional and independently controlled via Master.conf.
|
||||||
|
# Severity levels: normal, warning, alert
|
||||||
# Usage: notify "message" "subject" "severity"
|
# Usage: notify "message" "subject" "severity"
|
||||||
# Severity: normal, warning, alert
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
notify() {
|
notify() {
|
||||||
local message="$1"
|
local message="$1"
|
||||||
@@ -175,6 +199,7 @@ notify() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# DURATION FORMATTER
|
# DURATION FORMATTER
|
||||||
|
# Converts raw seconds into a human readable string — e.g. 10m53s or 47s
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
format_duration() {
|
format_duration() {
|
||||||
local secs=$1
|
local secs=$1
|
||||||
@@ -185,6 +210,10 @@ format_duration() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# ARG PARSER
|
# ARG PARSER
|
||||||
|
# Processes all flags and key=value pairs passed to any script.
|
||||||
|
# Supported flags: --dry-run, --log, --no-log, --status, --help
|
||||||
|
# Supported key=value: LOG=true/false, or any declared variable e.g. BW_LIMIT=5000
|
||||||
|
# Unparsed positional args returned in PARSED_ARGS array.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
parse_args() {
|
parse_args() {
|
||||||
ENABLE_LOGGING=${ENABLE_LOGGING:-false}
|
ENABLE_LOGGING=${ENABLE_LOGGING:-false}
|
||||||
@@ -229,12 +258,17 @@ parse_args() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# VALIDATION
|
# VALIDATION HELPERS
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
# Exits with error if a required variable is empty or unset.
|
||||||
|
# Usage: require_var VAR_NAME
|
||||||
require_var() {
|
require_var() {
|
||||||
[[ -z "${!1:-}" ]] && error "Missing required: $1" && exit 1
|
[[ -z "${!1:-}" ]] && error "Missing required: $1" && exit 1
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Exits with error if a variable is not a valid positive integer.
|
||||||
|
# Usage: validate_int VAR_NAME "$VAR_VALUE"
|
||||||
validate_int() {
|
validate_int() {
|
||||||
local name="$1" value="$2"
|
local name="$1" value="$2"
|
||||||
if [[ -z "$value" ]]; then
|
if [[ -z "$value" ]]; then
|
||||||
@@ -250,6 +284,9 @@ validate_int() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# HOST DETECTION
|
# HOST DETECTION
|
||||||
|
# Determines which server is local and which is remote by comparing hostname against
|
||||||
|
# HOST1 and HOST2 in Master.conf. Sets LOCAL_SERVER_NAME, REMOTE_SERVER_NAME and SSH_KEY.
|
||||||
|
# Both servers run identical scripts — this is what makes them bidirectional.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
detect_hosts() {
|
detect_hosts() {
|
||||||
LOCAL_HOSTNAME="$(hostname)"
|
LOCAL_HOSTNAME="$(hostname)"
|
||||||
@@ -271,12 +308,14 @@ detect_hosts() {
|
|||||||
SSH_KEY="${SSH_KEYS[$LOCAL_SERVER_NAME|$REMOTE_SERVER_NAME]}"
|
SSH_KEY="${SSH_KEYS[$LOCAL_SERVER_NAME|$REMOTE_SERVER_NAME]}"
|
||||||
|
|
||||||
[[ -z "$SSH_KEY" ]] && error "Missing SSH key mapping" && exit 1
|
[[ -z "$SSH_KEY" ]] && error "Missing SSH key mapping" && exit 1
|
||||||
|
|
||||||
info "$ICON_HOST Host: $LOCAL_SERVER_NAME → $REMOTE_SERVER_NAME"
|
info "$ICON_HOST Host: $LOCAL_SERVER_NAME → $REMOTE_SERVER_NAME"
|
||||||
}
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# REMOTE IP RESOLUTION
|
# REMOTE IP RESOLUTION
|
||||||
|
# Resolves the Tailscale IPv4 address of the remote server.
|
||||||
|
# Sets REMOTE_SERVER used by all subsequent SSH and rsync calls.
|
||||||
|
# Exits if resolution fails — Tailscale may be down or peer offline.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
resolve_remote_ip() {
|
resolve_remote_ip() {
|
||||||
log "Resolving remote IP for $REMOTE_SERVER_NAME..."
|
log "Resolving remote IP for $REMOTE_SERVER_NAME..."
|
||||||
@@ -287,6 +326,8 @@ resolve_remote_ip() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# CONNECTIVITY CHECK — fatal, used by rsync scripts
|
# CONNECTIVITY CHECK — fatal, used by rsync scripts
|
||||||
|
# Pings remote and exits if unreachable.
|
||||||
|
# For failover use ping_remote() which returns status without exiting.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_connectivity() {
|
check_connectivity() {
|
||||||
log "Checking connectivity to $REMOTE_SERVER..."
|
log "Checking connectivity to $REMOTE_SERVER..."
|
||||||
@@ -300,15 +341,15 @@ check_connectivity() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# PING REMOTE — non-fatal, used by failover
|
# PING REMOTE — non-fatal, used by failover
|
||||||
# Returns 0 if reachable, 1 if not
|
# Returns 0 if reachable, 1 if not — does NOT exit.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
ping_remote() {
|
ping_remote() {
|
||||||
ping -c2 -W3 "$REMOTE_SERVER" &>/dev/null
|
ping -c2 -W3 "$REMOTE_SERVER" &>/dev/null
|
||||||
}
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# PING INTERNET — non-fatal, used by failover
|
# PING INTERNET — non-fatal external connectivity check
|
||||||
# Returns 0 if internet reachable, 1 if not
|
# Returns 0 if internet reachable, 1 if not — does NOT exit.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
ping_internet() {
|
ping_internet() {
|
||||||
ping -c2 -W3 "${EXTERNAL_IP:-8.8.8.8}" &>/dev/null
|
ping -c2 -W3 "${EXTERNAL_IP:-8.8.8.8}" &>/dev/null
|
||||||
@@ -316,7 +357,9 @@ ping_internet() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# LOCAL ARRAY CHECK — non-fatal, returns status
|
# LOCAL ARRAY CHECK — non-fatal, returns status
|
||||||
# Used by failover before starting remote containers locally
|
# Verifies local /mnt/user is mounted and has shares.
|
||||||
|
# Used by failover before starting remote containers locally.
|
||||||
|
# Returns 0 if healthy, 1 if not.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_local_array() {
|
check_local_array() {
|
||||||
log "Checking local array..."
|
log "Checking local array..."
|
||||||
@@ -336,7 +379,9 @@ check_local_array() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# REMOTE ARRAY CHECK — non-fatal, returns status
|
# REMOTE ARRAY CHECK — non-fatal, returns status
|
||||||
# Used before handback rsync
|
# Verifies remote /mnt/user is mounted via SSH.
|
||||||
|
# Used before handback rsync — syncing to remote with no array fills rootfs.
|
||||||
|
# Returns 0 if healthy, 1 if not.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_remote_array() {
|
check_remote_array() {
|
||||||
log "Checking remote array on $REMOTE_SERVER_NAME..."
|
log "Checking remote array on $REMOTE_SERVER_NAME..."
|
||||||
@@ -353,7 +398,9 @@ check_remote_array() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# REMOTE DOCKER CHECK — non-fatal, returns status
|
# REMOTE DOCKER CHECK — non-fatal, returns status
|
||||||
# Used before starting containers on remote
|
# Verifies remote Docker daemon is responding before container operations.
|
||||||
|
# A hung daemon means start/stop commands will silently fail.
|
||||||
|
# Returns 0 if healthy, 1 if not.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_remote_docker() {
|
check_remote_docker() {
|
||||||
log "Checking remote Docker daemon on $REMOTE_SERVER_NAME..."
|
log "Checking remote Docker daemon on $REMOTE_SERVER_NAME..."
|
||||||
@@ -368,28 +415,26 @@ check_remote_docker() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# REMOTE ROOTFS SPACE CHECK — fatal
|
# REMOTE ROOTFS SPACE CHECK — fatal
|
||||||
|
# Aborts if remote rootfs exceeds ROOTFS_WARN threshold.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_remote_rootfs() {
|
check_remote_rootfs() {
|
||||||
log "Checking remote rootfs usage..."
|
log "Checking remote rootfs usage..."
|
||||||
REMOTE_USAGE=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
REMOTE_USAGE=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
||||||
"df / --output=pcent | tail -1 | tr -d ' %'" 2>/dev/null)
|
"df / --output=pcent | tail -1 | tr -d ' %'" 2>/dev/null)
|
||||||
|
|
||||||
if [[ -z "$REMOTE_USAGE" ]]; then
|
if [[ -z "$REMOTE_USAGE" ]]; then
|
||||||
error "Could not retrieve rootfs usage from $REMOTE_SERVER_NAME"
|
error "Could not retrieve rootfs usage from $REMOTE_SERVER_NAME"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "$REMOTE_USAGE" -ge "${ROOTFS_WARN:-75}" ]]; then
|
if [[ "$REMOTE_USAGE" -ge "${ROOTFS_WARN:-75}" ]]; then
|
||||||
error "$ICON_HEALTH Remote rootfs ${REMOTE_USAGE}% — threshold ${ROOTFS_WARN:-75}%"
|
error "$ICON_HEALTH Remote rootfs ${REMOTE_USAGE}% — threshold ${ROOTFS_WARN:-75}%"
|
||||||
warn "Array may be down or drives missing on $REMOTE_SERVER_NAME"
|
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
info "$ICON_HEALTH Remote rootfs: ${REMOTE_USAGE}% (threshold: ${ROOTFS_WARN:-75}%)"
|
info "$ICON_HEALTH Remote rootfs: ${REMOTE_USAGE}% (threshold: ${ROOTFS_WARN:-75}%)"
|
||||||
}
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# REMOTE SHARE VALIDATION — fatal
|
# REMOTE SHARE VALIDATION — fatal
|
||||||
|
# Verifies target directory exists and is not empty on remote.
|
||||||
# Usage: check_remote_share "/mnt/user/Movies"
|
# Usage: check_remote_share "/mnt/user/Movies"
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_remote_share() {
|
check_remote_share() {
|
||||||
@@ -398,7 +443,6 @@ check_remote_share() {
|
|||||||
|
|
||||||
SHARE_EXISTS=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
SHARE_EXISTS=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
||||||
"[[ -d '$dir' ]] && echo yes || echo no" 2>/dev/null)
|
"[[ -d '$dir' ]] && echo yes || echo no" 2>/dev/null)
|
||||||
|
|
||||||
if [[ "$SHARE_EXISTS" != "yes" ]]; then
|
if [[ "$SHARE_EXISTS" != "yes" ]]; then
|
||||||
error "$ICON_HEALTH Remote share does not exist: $dir"
|
error "$ICON_HEALTH Remote share does not exist: $dir"
|
||||||
exit 1
|
exit 1
|
||||||
@@ -406,7 +450,6 @@ check_remote_share() {
|
|||||||
|
|
||||||
SHARE_EMPTY=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
SHARE_EMPTY=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
||||||
"[[ -z \"\$(ls -A '$dir' 2>/dev/null)\" ]] && echo yes || echo no" 2>/dev/null)
|
"[[ -z \"\$(ls -A '$dir' 2>/dev/null)\" ]] && echo yes || echo no" 2>/dev/null)
|
||||||
|
|
||||||
if [[ "$SHARE_EMPTY" == "yes" ]]; then
|
if [[ "$SHARE_EMPTY" == "yes" ]]; then
|
||||||
warn "$ICON_HEALTH Remote share exists but is empty: $dir — aborting to protect data"
|
warn "$ICON_HEALTH Remote share exists but is empty: $dir — aborting to protect data"
|
||||||
exit 1
|
exit 1
|
||||||
@@ -417,6 +460,8 @@ check_remote_share() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# REMOTE DISK CHECK — fatal
|
# REMOTE DISK CHECK — fatal
|
||||||
|
# Verifies all physical disks backing a share are online on the remote server.
|
||||||
|
# Skipped when PROFILE_SKIP_DISK_CHECK is true (ZFS pools have no /mnt/disk* structure).
|
||||||
# Usage: check_remote_disks "/mnt/user/Movies"
|
# Usage: check_remote_disks "/mnt/user/Movies"
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
check_remote_disks() {
|
check_remote_disks() {
|
||||||
@@ -439,10 +484,8 @@ check_remote_disks() {
|
|||||||
local disk_mount disk_name
|
local disk_mount disk_name
|
||||||
disk_mount=$(dirname "$disk_share_path")
|
disk_mount=$(dirname "$disk_share_path")
|
||||||
disk_name=$(basename "$disk_mount")
|
disk_name=$(basename "$disk_mount")
|
||||||
|
|
||||||
MOUNTED=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
MOUNTED=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
||||||
"mountpoint -q '$disk_mount' && echo yes || echo no" 2>/dev/null)
|
"mountpoint -q '$disk_mount' && echo yes || echo no" 2>/dev/null)
|
||||||
|
|
||||||
if [[ "$MOUNTED" == "yes" ]]; then
|
if [[ "$MOUNTED" == "yes" ]]; then
|
||||||
info "$ICON_DISK $disk_name $ICON_RUNNING — $share_name present"
|
info "$ICON_DISK $disk_name $ICON_RUNNING — $share_name present"
|
||||||
else
|
else
|
||||||
@@ -455,12 +498,13 @@ check_remote_disks() {
|
|||||||
error "One or more disks backing $share_name are offline on $REMOTE_SERVER_NAME"
|
error "One or more disks backing $share_name are offline on $REMOTE_SERVER_NAME"
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
success "All disks backing $share_name are online"
|
success "All disks backing $share_name are online"
|
||||||
}
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# CONTAINER MANAGEMENT — STOP (remote via SSH)
|
# CONTAINER MANAGEMENT — STOP (remote via SSH)
|
||||||
|
# Stops containers in CRITICAL_CONTAINER_NAMES on remote server.
|
||||||
|
# Tracks running containers in RUNNING_CONTAINERS for restart after rsync.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
RUNNING_CONTAINERS=()
|
RUNNING_CONTAINERS=()
|
||||||
|
|
||||||
@@ -470,15 +514,12 @@ stop_containers() {
|
|||||||
log "No containers configured for this profile, skipping stop."
|
log "No containers configured for this profile, skipping stop."
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
|
|
||||||
info "Stopping containers..."
|
info "Stopping containers..."
|
||||||
RUNNING_CONTAINERS=()
|
RUNNING_CONTAINERS=()
|
||||||
|
|
||||||
for c in "${CRITICAL_CONTAINER_NAMES[@]}"; do
|
for c in "${CRITICAL_CONTAINER_NAMES[@]}"; do
|
||||||
[[ -z "$c" ]] && continue
|
[[ -z "$c" ]] && continue
|
||||||
STATUS=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
STATUS=$(ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" \
|
||||||
"docker inspect -f '{{.State.Running}}' $c 2>/dev/null" 2>/dev/null || echo "false")
|
"docker inspect -f '{{.State.Running}}' $c 2>/dev/null" 2>/dev/null || echo "false")
|
||||||
|
|
||||||
if [[ "$STATUS" == "true" ]]; then
|
if [[ "$STATUS" == "true" ]]; then
|
||||||
echo "$ICON_STOP Stopping $c..."
|
echo "$ICON_STOP Stopping $c..."
|
||||||
RUNNING_CONTAINERS+=("$c")
|
RUNNING_CONTAINERS+=("$c")
|
||||||
@@ -495,28 +536,25 @@ stop_containers() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# CONTAINER MANAGEMENT — START (remote via SSH)
|
# CONTAINER MANAGEMENT — START (remote via SSH)
|
||||||
|
# Restarts only containers tracked in RUNNING_CONTAINERS.
|
||||||
|
# Delayed containers receive CONTAINER_DELAY seconds before starting.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
start_containers() {
|
start_containers() {
|
||||||
if [[ ${#RUNNING_CONTAINERS[@]} -eq 0 ]]; then
|
if [[ ${#RUNNING_CONTAINERS[@]} -eq 0 ]]; then
|
||||||
log "No containers to restart."
|
log "No containers to restart."
|
||||||
return
|
return
|
||||||
fi
|
fi
|
||||||
|
|
||||||
info "Starting containers..."
|
info "Starting containers..."
|
||||||
|
|
||||||
for c in "${RUNNING_CONTAINERS[@]}"; do
|
for c in "${RUNNING_CONTAINERS[@]}"; do
|
||||||
[[ -z "$c" ]] && continue
|
[[ -z "$c" ]] && continue
|
||||||
|
|
||||||
local needs_delay=false
|
local needs_delay=false
|
||||||
for d in "${DELAYED_CONTAINERS[@]}"; do
|
for d in "${DELAYED_CONTAINERS[@]}"; do
|
||||||
[[ "$c" == "$d" ]] && needs_delay=true && break
|
[[ "$c" == "$d" ]] && needs_delay=true && break
|
||||||
done
|
done
|
||||||
|
|
||||||
if [[ "$needs_delay" == true ]]; then
|
if [[ "$needs_delay" == true ]]; then
|
||||||
info "Waiting ${CONTAINER_DELAY}s before starting $c..."
|
info "Waiting ${CONTAINER_DELAY}s before starting $c..."
|
||||||
sleep "$CONTAINER_DELAY"
|
sleep "$CONTAINER_DELAY"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "$ICON_START Starting $c..."
|
echo "$ICON_START Starting $c..."
|
||||||
if ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" "docker start $c" >/dev/null; then
|
if ssh -i "$SSH_KEY" root@"$REMOTE_SERVER" "docker start $c" >/dev/null; then
|
||||||
echo "$ICON_STARTED $c started"
|
echo "$ICON_STARTED $c started"
|
||||||
@@ -528,6 +566,8 @@ start_containers() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# RSYNC OPTIONS
|
# RSYNC OPTIONS
|
||||||
|
# Loads rsync options for current profile. Falls back to DEFAULT_RSYNC_OPTS if no match.
|
||||||
|
# Profile opts do NOT inherit from defaults — list all desired flags explicitly.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
get_rsync_opts() {
|
get_rsync_opts() {
|
||||||
if [[ -n "${PROFILE_RSYNC_OPTS[$PROFILE_NAME]:-}" ]]; then
|
if [[ -n "${PROFILE_RSYNC_OPTS[$PROFILE_NAME]:-}" ]]; then
|
||||||
@@ -541,6 +581,7 @@ get_rsync_opts() {
|
|||||||
|
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
# STATUS DISPLAY
|
# STATUS DISPLAY
|
||||||
|
# Prints current runtime configuration — triggered by --status flag.
|
||||||
# -----------------------------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------------------------
|
||||||
show_status() {
|
show_status() {
|
||||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||||
|
|||||||
+86
-56
@@ -7,30 +7,26 @@
|
|||||||
# Copy and paste the contents of this file into a new User Script entry in the plugin,
|
# Copy and paste the contents of this file into a new User Script entry in the plugin,
|
||||||
# then uncomment the script you want to run and set your schedule.
|
# then uncomment the script you want to run and set your schedule.
|
||||||
#
|
#
|
||||||
# All scripts in this ecosystem live at:
|
# All scripts live at: /mnt/user/appdata/unraid_scripts/
|
||||||
# /mnt/user/appdata/unraid_scripts/
|
# All configuration at: /mnt/user/appdata/unraid_scripts/Master.conf
|
||||||
#
|
# Shared functions at: /mnt/user/appdata/unraid_scripts/common.sh
|
||||||
# Configuration for all scripts is managed in one place:
|
|
||||||
# /mnt/user/appdata/unraid_scripts/Master.conf
|
|
||||||
#
|
|
||||||
# Shared runtime functions used by all scripts:
|
|
||||||
# /mnt/user/appdata/unraid_scripts/common.sh
|
|
||||||
#
|
#
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
# Changelog:
|
# Changelog:
|
||||||
# v1.0 — Initial template
|
# v1.0 — Initial template
|
||||||
# v1.1 — MAX_RSYNC_PROCS removed — bandwidth limiting handles concurrency
|
# v1.1 — MAX_RSYNC_PROCS removed, --status flag, full directory tree, changelog
|
||||||
# Typo fixes, --status flag added, full directory tree, changelog added
|
|
||||||
# v1.2 — Docker Essentials, Media, Transcodes, System Watchdog added
|
# v1.2 — Docker Essentials, Media, Transcodes, System Watchdog added
|
||||||
|
# v1.3 — Failover script added
|
||||||
|
# v1.4 — WebGUI watchdog, ZFS snapshot, Docker network connect, schedules section
|
||||||
|
# v1.5 — media_management.sh orchestrator, corrected filenames, git repo section
|
||||||
|
# v1.6 — lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh added
|
||||||
|
# v1.7 — cert_monitor.sh added
|
||||||
|
# v1.8 — Monitor/ folder added with all monitoring scripts
|
||||||
|
# cert_monitor.sh moved from unRAID_Essentials to Monitor/
|
||||||
|
# backup_verify.sh, smart_health.sh, bandwidth_monitor.sh added
|
||||||
|
# weekly_health_digest.sh, emby_session_report.sh added
|
||||||
|
# ZFS memory snapshot moved to Monitor/ — informational only
|
||||||
# Directory tree updated to reflect full ecosystem
|
# Directory tree updated to reflect full ecosystem
|
||||||
# v1.3 — Failover script added, directory tree updated with Failover folder
|
|
||||||
# v1.4 — WebGUI watchdog, ZFS memory snapshot, Docker network connect added
|
|
||||||
# Recommended schedules section added
|
|
||||||
# v1.5 — media_management.sh orchestrator added to Orchestrators folder
|
|
||||||
# Corrected filenames: media_shares_permissions.sh, user_scripts_stop.sh
|
|
||||||
# Git repo section added
|
|
||||||
# Individual media cleaner entries replaced by media_management.sh
|
|
||||||
# Directory tree updated
|
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
|
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
@@ -47,9 +43,18 @@
|
|||||||
# ├── Failover/
|
# ├── Failover/
|
||||||
# │ └── failover.sh # Mutual container failover — runs continuously
|
# │ └── failover.sh # Mutual container failover — runs continuously
|
||||||
# │
|
# │
|
||||||
|
# ├── Monitor/
|
||||||
|
# │ ├── backup_verify.sh # Random sample checksum verification vs remote
|
||||||
|
# │ ├── cert_monitor.sh # SSL certificate expiry — direct openssl check
|
||||||
|
# │ ├── emby_session_report.sh # Weekly Emby usage statistics via API
|
||||||
|
# │ ├── smart_health.sh # Drive SMART attribute monitoring
|
||||||
|
# │ ├── bandwidth_monitor.sh # Daily transfer logging + weekly summary
|
||||||
|
# │ ├── weekly_health_digest.sh # Aggregated system health — always/smart/weekly
|
||||||
|
# │ └── zfs_memory_snapshot.sh # Weekly ZFS health and memory diagnostic report
|
||||||
|
# │
|
||||||
# ├── Orchestrators/
|
# ├── Orchestrators/
|
||||||
# │ ├── daily_sync.sh # Runs all daily media share syncs sequentially
|
# │ ├── daily_sync.sh # Runs all daily media share syncs sequentially
|
||||||
# │ └── media_management.sh # Runs permissions + anime + media cleaner sequentially
|
# │ └── media_management.sh # Runs permissions + cleaners + arr cleanup sequentially
|
||||||
# │
|
# │
|
||||||
# ├── Rsync/
|
# ├── Rsync/
|
||||||
# │ ├── rsync.sh # Core rsync script — called per share or profile
|
# │ ├── rsync.sh # Core rsync script — called per share or profile
|
||||||
@@ -63,11 +68,14 @@
|
|||||||
# │
|
# │
|
||||||
# ├── Media/
|
# ├── Media/
|
||||||
# │ ├── media_shares_permissions.sh # Applies permissions to all media shares
|
# │ ├── media_shares_permissions.sh # Applies permissions to all media shares
|
||||||
# │ └── media_cleaner.sh # Removes junk files — profiles: anime, media
|
# │ ├── media_cleaner.sh # Removes junk files — profiles: anime, media
|
||||||
|
# │ ├── lidarr_cleanup.sh # Removes orphaned music files via Lidarr API
|
||||||
|
# │ ├── sonarr_cleanup.sh # Removes orphaned TV files via Sonarr API
|
||||||
|
# │ └── radarr_cleanup.sh # Removes orphaned movie files via Radarr API
|
||||||
# │
|
# │
|
||||||
# ├── Transcodes/
|
# ├── Transcodes/
|
||||||
# │ ├── ramdisk_setup.sh # Creates ramdisk and transcode symlink — run at array start
|
# │ ├── ramdisk_setup.sh # Creates ramdisk and transcode symlink
|
||||||
# │ ├── transcode_manager.sh # Monitors ramdisk usage, manages symlink direction
|
# │ ├── transcode_manager.sh # Monitors ramdisk usage, manages symlink
|
||||||
# │ └── transcode_cleanup.sh # Removes old inactive transcode files
|
# │ └── transcode_cleanup.sh # Removes old inactive transcode files
|
||||||
# │
|
# │
|
||||||
# ├── Tools/
|
# ├── Tools/
|
||||||
@@ -82,34 +90,39 @@
|
|||||||
# ├── server_reboot.sh # Graceful server reboot with user warning
|
# ├── server_reboot.sh # Graceful server reboot with user warning
|
||||||
# ├── system_watchdog.sh # System health monitor — last line of defense
|
# ├── system_watchdog.sh # System health monitor — last line of defense
|
||||||
# ├── user_scripts_stop.sh # Stops running User Scripts plugin jobs
|
# ├── user_scripts_stop.sh # Stops running User Scripts plugin jobs
|
||||||
# ├── webgui_restart.sh # WebGUI watchdog — nginx + emhttp escalating restart
|
# └── webgui_restart.sh # WebGUI watchdog — nginx + emhttp restart
|
||||||
# └── zfs_memory_snapshot.sh # Weekly ZFS health and memory diagnostic report
|
|
||||||
#
|
#
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
# ━━━ 🚀 Script Commands — Uncomment the one you want to run ━━━
|
# ━━━ 🚀 Script Commands — Uncomment the one you want to run ━━━
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
#
|
#
|
||||||
# ━━━ Failover (README) ━━━
|
# ━━━ Failover (README) ━━━
|
||||||
# Runs continuously as a background task — do NOT schedule with a time interval.
|
# Runs continuously as background task — set "At Startup of Array" as background task.
|
||||||
# Set schedule to "At Startup of Array" — background task.
|
# Both servers must have this running for mutual failover to work.
|
||||||
# Both servers must have this script running for mutual failover to work.
|
|
||||||
# Use --status to check current state without restarting the loop.
|
|
||||||
#
|
#
|
||||||
# ━━━ Failover ━━━
|
|
||||||
#/mnt/user/appdata/unraid_scripts/Failover/failover.sh
|
#/mnt/user/appdata/unraid_scripts/Failover/failover.sh
|
||||||
#
|
#
|
||||||
|
# ━━━ Monitor ━━━
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/backup_verify.sh
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/cert_monitor.sh
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/emby_session_report.sh
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/smart_health.sh
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/bandwidth_monitor.sh
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/weekly_health_digest.sh
|
||||||
|
#/mnt/user/appdata/unraid_scripts/Monitor/zfs_memory_snapshot.sh
|
||||||
|
#
|
||||||
# ━━━ Orchestrators ━━━
|
# ━━━ Orchestrators ━━━
|
||||||
#/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
|
#/mnt/user/appdata/unraid_scripts/Orchestrators/daily_sync.sh
|
||||||
#/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
|
#/mnt/user/appdata/unraid_scripts/Orchestrators/media_management.sh
|
||||||
#
|
#
|
||||||
# ━━━ Rsync — Appdata Profiles (scheduled individually) ━━━
|
# ━━━ Rsync — Appdata Profiles ━━━
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Arrs_Stack
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Critical-Data
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Critical-Data
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Important-Data
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Important-Data
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Media_Server/Emby
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Emby
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Gmer4Lfe
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/appdata-Failover/Gmer4Lfe
|
||||||
#
|
#
|
||||||
# ━━━ Rsync — Individual Media Shares (ad hoc use) ━━━
|
# ━━━ Rsync — Individual Media Shares (ad hoc) ━━━
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Movies
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Movies
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Tv_Shows
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Tv_Shows
|
||||||
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Anime_Shows
|
#/mnt/user/appdata/unraid_scripts/Rsync/rsync.sh /mnt/user/Anime_Shows
|
||||||
@@ -148,51 +161,68 @@
|
|||||||
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh
|
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/system_watchdog.sh
|
||||||
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/user_scripts_stop.sh
|
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/user_scripts_stop.sh
|
||||||
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh
|
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/webgui_restart.sh
|
||||||
#/mnt/user/appdata/unraid_scripts/unRAID_Essentials/zfs_memory_snapshot.sh
|
|
||||||
#
|
#
|
||||||
# ━━━ Git repo (Gitea) ━━━
|
# ━━━ Git repo (Gitea) ━━━
|
||||||
#/mnt/user/appdata/unraid_scripts/git_pull_execute.sh
|
#/mnt/user/appdata/unraid_scripts/git_pull_execute.sh
|
||||||
#
|
#
|
||||||
|
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
# ━━━ ⚙️ Arguments — Add after the script path ━━━
|
# ━━━ ⚙️ Arguments ━━━
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
#
|
#
|
||||||
# --dry-run Preview what would happen — no changes made
|
# --dry-run Preview what would happen — no changes made
|
||||||
# --log Enable verbose logging output
|
# --log Enable verbose logging output
|
||||||
# --no-log Disable logging (overrides Master.conf setting)
|
# --no-log Disable logging (overrides Master.conf)
|
||||||
# --status Print resolved configuration and current state then exit
|
# --status Print resolved config and current state then exit
|
||||||
# --help Show usage information
|
# --help Show usage information
|
||||||
# KEY=VALUE Override any Master.conf variable for this run only
|
# KEY=VALUE Override any Master.conf variable for this run only
|
||||||
#
|
#
|
||||||
# ━━━ Failover specific ━━━
|
# ━━━ Special arguments ━━━
|
||||||
# failover.sh --status — check current state
|
# failover.sh --status — check current state without restarting loop
|
||||||
# failover.sh --dry-run --log — test logic without touching containers
|
# failover.sh --dry-run --log — test logic without touching containers
|
||||||
|
# media_cleaner.sh anime --dry-run — profile required as first argument
|
||||||
|
# bandwidth_monitor.sh --report — generate weekly summary report
|
||||||
|
# lidarr/sonarr/radarr --dry-run --log — always test arr cleanup first
|
||||||
#
|
#
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
# ━━━ 📋 Recommended Schedules ━━━
|
# ━━━ 📋 Recommended Schedules ━━━
|
||||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
#
|
#
|
||||||
# failover.sh — At Startup of Array (background task)
|
# ━━━ At Startup of Array ━━━
|
||||||
# ramdisk_setup.sh — At Startup of Array
|
# failover.sh — background task
|
||||||
# docker_network_connect.sh — At Startup of Array
|
# ramdisk_setup.sh
|
||||||
# docker_syslog_filter.sh — At Startup of Array
|
# docker_network_connect.sh
|
||||||
# php_fpm_max_children.sh — At Startup of Array
|
# docker_syslog_filter.sh
|
||||||
|
# php_fpm_max_children.sh
|
||||||
#
|
#
|
||||||
# transcode_manager.sh — */3 * * * * (every 3 minutes)
|
# ━━━ Frequent (every few minutes) ━━━
|
||||||
# transcode_cleanup.sh — */5 * * * * (every 5 minutes)
|
# */3 * * * * transcode_manager.sh
|
||||||
# docker_watchdog.sh — */15 * * * * (every 15 minutes)
|
# */5 * * * * transcode_cleanup.sh
|
||||||
# webgui_restart.sh — */10 * * * * (every 10 minutes)
|
# */10 * * * * webgui_restart.sh
|
||||||
# system_watchdog.sh — */15 * * * * (every 15 minutes)
|
# */15 * * * * docker_watchdog.sh
|
||||||
|
# */15 * * * * system_watchdog.sh
|
||||||
#
|
#
|
||||||
# daily_sync.sh — 0 1 * * * (daily at 1am)
|
# ━━━ Daily ━━━
|
||||||
# docker_daily_restart.sh — 0 3 * * * (daily at 3am)
|
# 0 1 * * * daily_sync.sh
|
||||||
# media_management.sh — 0 2 * * * (daily at 2am)
|
# 0 2 * * * media_management.sh
|
||||||
|
# 0 3 * * * docker_daily_restart.sh
|
||||||
|
# 0 8 * * * weekly_health_digest.sh (profile controls if it sends)
|
||||||
#
|
#
|
||||||
# clear_logs.sh — 0 5 * * 0 (weekly Sunday 5am)
|
# ━━━ Weekly — Sunday morning block ━━━
|
||||||
# zfs_memory_snapshot.sh — 0 6 * * 0 (weekly Sunday 6am)
|
# 0 3 * * 0 docker_weekly_restart.sh
|
||||||
# docker_weekly_restart.sh — 0 3 * * 0 (weekly Sunday 3am)
|
# 0 5 * * 0 clear_logs.sh
|
||||||
|
# 0 6 * * 0 zfs_memory_snapshot.sh
|
||||||
|
# 0 7 * * 0 smart_health.sh
|
||||||
|
# 0 8 * * 0 weekly_health_digest.sh (weekly profile sends today)
|
||||||
|
# 0 9 * * 0 cert_monitor.sh
|
||||||
|
# 0 10 * * 0 backup_verify.sh
|
||||||
|
# 0 11 * * 0 emby_session_report.sh
|
||||||
#
|
#
|
||||||
# rsync.sh profiles — schedule individually as needed
|
# ━━━ Rsync profiles ━━━
|
||||||
|
# Schedule individually as needed
|
||||||
|
#
|
||||||
|
# ━━━ Bandwidth monitor ━━━
|
||||||
|
# bandwidth_monitor.sh --log-transfer is called automatically by rsync.sh
|
||||||
|
# bandwidth_monitor.sh --report runs standalone for weekly summary
|
||||||
#
|
#
|
||||||
# ==============================================================================================
|
# ==============================================================================================
|
||||||
Reference in New Issue
Block a user