53 KiB
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📊 MONITORS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Observe, measure, and report — never act. Scripts in this folder watch the system and notify when something needs attention. They do not restart containers, do not reboot servers, do not delete files. The other folders handle intervention. This folder handles awareness.
Monitors/ ← observes and reports (this folder)
Docker_Essentials/ ← acts on containers
unRAID_Essentials/ ← acts on the server
Media/ ← maintains the library
All monitor scripts are safe to run at any time. They are read-only except for
bandwidth_monitor.sh(one bounded append per rsync run) andsystem_tuning_monitor.sh(one bounded append every 6 hours). Every other script in this folder writes nothing. None of them can break anything.
━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔴 Certs Expiring Silently Despite Renewal
Certbot runs on schedule. NPM shows the cert as renewed. But the web server was not
reloaded after renewal — the old expired cert is still being served to external clients.
NPM's dashboard shows green. openssl s_client from the outside shows a cert that
expired three days ago. Users see certificate errors. You see a healthy NPM status.
The fix: cert_monitor.sh connects directly via openssl — not to NPM's API, not
to any internal check, but to the actual TLS handshake the outside world sees. It
catches renewed-but-not-reloaded certs, wrong certs being served, chain issues that
are invisible from the inside. If users would see an error, this script catches it.
🔴 No Early Warning on Drive Degradation
A drive accumulates reallocated sectors silently over weeks. The count climbs from 0 to 12 to 47. Nothing fails. No error in the Docker logs. No alert from unRAID. The drive is in progressive failure — heading toward a point of no return — and the only way to know is to actively query SMART data. Which nobody does until something goes wrong.
The fix: smart_health.sh queries SMART attributes for every drive every week.
Reallocated sectors, pending sectors, uncorrectable sectors, temperature — anything
concerning triggers a notification. Drive failure has a warning curve. Weekly SMART
checks give you time to act on it.
🔴 rsync Succeeds But the Copy Is Wrong
rsync reports success. Transfer completed. Exit code 0. But a storage hardware
issue on either server silently corrupted a file during the write. The file exists
on both ends. rsync is satisfied — sizes match, timestamps match. Nobody knows the
content is wrong until someone tries to play the file and it fails, or worse, until
a drive failure reveals the backup was corrupt from the start.
The fix: backup_verify.sh independently computes MD5 checksums on both sides of
the rsync mirror and compares them. It does not trust rsync's exit code. It does not
trust that the file exists. It verifies the actual content matches. Silent data
corruption is the failure mode backup systems exist to catch — and the only way to
catch it is to verify independently.
🔴 inotify Exhaustion Producing Unexplained Symptoms
inotify watches are silently exhausted. The kernel limit is hit. New file events stop being delivered — silently. Downloads finish but arr containers don't see them. Live TV stutter appears because media files aren't being detected as they arrive. Library updates stop happening. None of these produce clear error messages — the containers are running, the files are there, nothing is obviously wrong. The symptom looks like a network problem or a container bug. It is an inotify problem.
The same issue appears with PHP-FPM worker exhaustion — WebGUI requests start queueing, timeouts appear, the dashboard feels sluggish. The worker count is at ceiling and you have no visibility into how often it happens or how close it gets.
The fix: system_tuning_monitor.sh snapshots inotify and PHP-FPM utilisation every
6 hours. The weekly coffee report shows peak, average, and warning count. You see
the trend before it becomes a problem. inotify_tuning.sh and php_fpm_max_children.sh
fix the limits — this script tells you whether the fix is holding.
🔴 No Single View of What the Continuous Scripts Are Doing
Three scripts run continuously as background processes: system_watchdog.sh,
docker_watchdog.sh, and failover.sh. Are they running? How long have they been
up? Are there active strikes against any container? Is failover in a non-NORMAL state?
Is the skip list empty? You have to check each state file individually, parse the
format yourself, and mentally correlate the information.
The fix: continuous_scripts_status.sh — a live dashboard that reads all state files
and presents a single coherent view. Run it any time to know exactly what the watchdogs
are doing, what strikes are active, what the failover state is, and how the system
health looks right now. One command, complete picture.
━━━ WHAT THIS FOLDER DOES ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Four distinct observation roles:
🔍 Live Connectivity Checks — Active probes against real endpoints
cert_monitor.sh and backup_verify.sh go out and test things. They connect,
compute, compare. The result is a direct measurement of the actual state of the
world — not a reading of a cached status.
📋 State File Aggregation — Synthesising the ecosystem's own data
weekly_health_digest.sh and continuous_scripts_status.sh read the state files
the watchdogs and orchestrators have already written. They don't gather new data —
they organise existing data into human-readable summaries.
📈 Time-Series Snapshots — Tracking trends over time
bandwidth_monitor.sh and system_tuning_monitor.sh write one bounded log entry
per interval. Over time this builds a picture of trends — whether inotify pressure
is growing, whether transfer sizes are increasing, whether PHP-FPM is consistently
near ceiling.
🖥️ Hardware Health — Checking what the hardware itself reports
smart_health.sh and zfs_memory_snapshot.sh query the hardware and kernel
directly — drive SMART attributes, ZFS pool state, ARC statistics, kernel memory
pressure. Things that are invisible in application-level monitoring.
━━━ FLASH DRIVE WRITE POLICY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
unRAID boots from a USB flash drive. Flash drives have limited write cycles. Every script in this folder is designed with this constraint in mind.
| Script | Writes to flash | Detail |
|---|---|---|
cert_monitor.sh |
Never | openssl connections only |
smart_health.sh |
Never | smartctl reads only |
zfs_memory_snapshot.sh |
Never | writes to /var/log/ (tmpfs) |
backup_verify.sh |
Never | SSH + MD5 comparison only |
emby_session_report.sh |
Never | API queries only |
weekly_health_digest.sh |
Never | reads existing state files only |
continuous_scripts_status.sh |
Never | reads existing state files only |
bandwidth_monitor.sh |
One append + one trim per rsync run | Bounded — never exceeds BANDWIDTH_LOG_RETENTION lines |
system_tuning_monitor.sh |
One append + one trim every 6 hours | Bounded — trimmed to TUNING_LOG_RETENTION days |
The two scripts that write to flash are designed to be minimal and bounded. They write one line and trim old entries on every write — the files never grow.
━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
| Script | What It Watches | When |
|---|---|---|
cert_monitor.sh |
SSL cert expiry via live TLS connection | Weekly — Sunday 9am |
smart_health.sh |
Drive SMART health attributes | Weekly — Sunday 7am |
zfs_memory_snapshot.sh |
ZFS pool health, ARC stats, memory pressure | Weekly — Sunday 6am |
backup_verify.sh |
rsync mirror integrity via MD5 checksums | Weekly — Sunday 10am |
bandwidth_monitor.sh |
rsync transfer history and trends | Auto (by rsync.sh) + weekly report |
weekly_health_digest.sh |
Full ecosystem health aggregation | Daily (profile controls notify) |
emby_session_report.sh |
Emby streaming usage and library stats | Weekly — Sunday 11am |
system_tuning_monitor.sh |
inotify + PHP-FPM utilisation trends | Every 6 hours |
continuous_scripts_status.sh |
Live dashboard for all background processes | On demand |
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🔐 cert_monitor.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SSL certificate expiry monitoring via live TLS connection. Connects directly to each
domain via openssl s_client and reads the certificate actually being served —
not the certificate file, not NPM's API, not any cached status.
# Scheduled: 0 9 * * 0 (Sunday 9am weekly)
── Why Direct openssl, Not an API ──────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# API-based cert checks ask the certificate manager if the cert is valid.
# openssl checks ask the server what cert it is actually serving.
# These are not the same question and the answers can differ.
#
# Direct openssl catches:
# - Cert renewed in NPM but web server not reloaded (old cert still serving)
# - Wrong cert being served (subdomain pointing to wrong vhost)
# - Chain issues visible externally but not internally
# - NPM reporting healthy while clients see an expired cert
#
# If a user would see a certificate error in their browser, this script
# catches it. If the cert manager says it's fine but the server is serving
# an old cert, this script catches that too.
── Results Per Domain ───────────────────────────────────────────────────────
HEALTHY > CERT_WARN_DAYS remaining → silent ✅
WARNING ≤ CERT_WARN_DAYS remaining → notification
CRITICAL ≤ CERT_CRIT_DAYS remaining → urgent notification
FAILED could not connect or parse cert → notification
Notifications are batched by severity — one message lists all domains at WARNING, a separate message lists all domains at CRITICAL. Not one notification per domain.
── Configuration ────────────────────────────────────────────────────────────
# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# Each server monitors its own domains — detect_hosts() aliases the correct list.
# Each domain and subdomain is a separate entry — they have independent certs.
#
HOST1_CERT_MONITOR_DOMAINS=(
"Gmer4Lfe.com" # primary domain — checks the actual TLS handshake
"Gmer4Lfe.us" # secondary domain
# "auth.Gmer4Lfe.com" # add subdomains separately — they have separate certs
# "cloud.Gmer4Lfe.com"
)
# master.conf
CERT_WARN_DAYS=30 # warn this many days before expiry — time to investigate
CERT_CRIT_DAYS=7 # critical alert — action needed now
CERT_TIMEOUT=10 # seconds per domain before declaring FAILED
── Usage ───────────────────────────────────────────────────────────────────
cert_monitor.sh # normal run — silent if all healthy
cert_monitor.sh --dry-run # check all certs and show results, no notification sent
cert_monitor.sh --log # verbose output per domain
cert_monitor.sh --status # show domain list and thresholds
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
💾 smart_health.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Drive SMART health monitoring. Queries SMART attributes for every drive on the system weekly. Catches drive degradation before it becomes drive failure — most spinning drives go through a warning period before dying outright.
# Scheduled: 0 7 * * 0 (Sunday 7am weekly)
── What It Monitors Per Drive ──────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# The attributes that actually indicate drive health problems:
#
Reallocated_Sector_Ct any > 0 = warning
The drive found a bad sector and swapped in a spare.
0 spare = healthy. 1 spare = warning. More = escalating concern.
This is the first sign a drive is wearing out.
Current_Pending_Sector any > 0 = warning
Sectors the drive suspects are bad but hasn't confirmed yet.
May recover on next read. May escalate to Reallocated. Watch it.
Offline_Uncorrectable any > 0 = critical
Sectors the drive tried to correct during offline tests and couldn't.
No spares. No recovery. Data loss risk.
Temperature_Celsius vs thresholds from dynamix.cfg
High sustained temperature shortens drive life significantly.
Script reads unRAID's own configured thresholds — no duplication needed.
Overall SMART status PASSED / FAILED
The drive's own self-assessment. FAILED = get data off this drive now.
Power_On_Hours informational
Drive age estimate — useful context for evaluating other attributes.
# ─────────────────────────────────────────────────────────────────────────────
── Drive Discovery + Temperature Source ────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Drives discovered automatically — /dev/sd* and /dev/nvme* on every run.
# NVMe drives use different attribute names — detected and handled automatically.
# No drive list to maintain in configuration.
#
# Temperature thresholds read from /boot/config/plugins/dynamix/dynamix.cfg:
# hot, max, hotssd, maxssd — unRAID's own configured values.
# Falls back to SMART_TEMP_WARN / SMART_TEMP_CRIT from master.conf if not found.
# This means smart_health.sh and unRAID's dashboard use the same thresholds.
── Configuration ────────────────────────────────────────────────────────────
# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# Drives to skip — typically the boot USB flash drive.
# unRAID boots from USB which appears as sda. Flash drives either don't
# support SMART or report meaningless values. Skip it.
#
HOST1_SMART_IGNORE_DRIVES=(
"sda" # boot USB flash drive — no meaningful SMART data
)
# master.conf
SMART_TEMP_WARN=45 # °C — fallback if dynamix.cfg not found
SMART_TEMP_CRIT=55 # °C — fallback
── Usage ───────────────────────────────────────────────────────────────────
smart_health.sh # normal run — silent if all drives healthy
smart_health.sh --dry-run # show which drives would be checked, no smartctl
smart_health.sh --log # verbose — full attribute dump per drive
smart_health.sh --status # show config and ignore list
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🗄️ zfs_memory_snapshot.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Weekly ZFS pool health and memory diagnostic report. Combines ZFS pool status, ARC
statistics, Docker memory usage, and kernel memory pressure into a single snapshot.
Informational only — system_watchdog.sh handles threshold-based intervention.
# Scheduled: 0 6 * * 0 (Sunday 6am weekly — before other Sunday scripts)
── What It Reports ─────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# ZFS pool health:
# Status (ONLINE/DEGRADED/FAULTED) per pool
# Size, allocated, free, capacity, health per pool
# Pools in ZFS_REPORT_IGNORE_POOLS are excluded from reporting
# (they remain fully monitored by unRAID — this only affects the report)
#
# ARC statistics:
# Current ARC size vs max — what percentage of max is being used
# Metadata ARC — how much ARC is occupied by metadata vs data
# ARC hit rate — cache effectiveness
# Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT
#
# Memory status:
# Total, free, available RAM
# Warns if free < ZFS_REPORT_FREE_WARN_GB
# Warns if available < ZFS_REPORT_AVAIL_WARN_GB
#
# Docker memory:
# Top ZFS_REPORT_DOCKER_TOP containers by memory usage
# Useful for spotting containers approaching their watchdog limits
#
# Kernel pressure:
# vmstat snapshot — 3 samples showing CPU and memory activity
# ─────────────────────────────────────────────────────────────────────────────
── Output Goes to Console and Log ─────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# Output is written to both console (for User Scripts output log) and ZFS_REPORT_LOG.
# The log file lets you compare pool health week over week without having to
# remember what last week's numbers were. Open it to see the trend.
# In dry-run mode — console only, nothing written to log.
#
ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log" # tmpfs — resets on reboot
── Configuration ────────────────────────────────────────────────────────────
# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# Single-disk ZFS array members can be excluded from the report.
# They remain fully monitored by unRAID — this only affects the weekly snapshot.
# Each disk in a JBOD ZFS configuration appears as its own pool — there can be many.
#
HOST1_ZFS_REPORT_IGNORE_POOLS=(
"disk10" # JBOD member — high usage expected, exclude from report noise
"disk9"
"disk8"
"disk6"
"disk5"
)
# master.conf
ZFS_REPORT_ARC_WARN_PCT=90 # warn if ARC using more than 90% of its max
ZFS_REPORT_FREE_WARN_GB=10 # warn if free RAM below 10GB
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if available RAM below 20GB
ZFS_REPORT_DOCKER_TOP=10 # top 10 Docker containers by memory
── Usage ───────────────────────────────────────────────────────────────────
zfs_memory_snapshot.sh # normal run — writes to log + console
zfs_memory_snapshot.sh --dry-run # console only, no log write
zfs_memory_snapshot.sh --log # verbose output
zfs_memory_snapshot.sh --status # show config and pool ignore list
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✅ backup_verify.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
rsync mirror integrity verification via independent MD5 checksums. Randomly samples files from each share, computes checksums on both sides, and compares. Does not trust rsync's exit code. Verifies the actual content matches.
# Scheduled: 0 10 * * 0 (Sunday 10am weekly)
── Why Verify Independently ─────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# rsync reports success when:
# - The transfer completed without network errors
# - The file sizes and modification times match
#
# rsync does NOT detect:
# - Silent corruption during the transfer (bitflip in transit)
# - Corruption written to storage at rest (faulty drive sector)
# - Files that matched size/mtime but had wrong content (rare, happens)
#
# These failure modes produce exit code 0. rsync says "done."
# backup_verify.sh checks whether "done" means "correct."
#
# Sample size is intentionally small (10 files per share, default) —
# this is a spot check, not an exhaustive verify. It catches systematic
# problems and hardware issues while running in minutes, not hours.
# ─────────────────────────────────────────────────────────────────────────────
── Results Per File ─────────────────────────────────────────────────────────
MATCH checksums identical on both sides → correctly mirrored ✅
MISMATCH file exists on both, checksums differ → sync failure or corruption
MISSING file exists locally but not on remote → not yet synced or deleted
All MISMATCHes and any significant number of MISSINGs trigger a notification.
── Pre-flight Checks Before Any SSH ───────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Two checks before computing a single checksum:
#
# check_connectivity() — is the remote Tailscale IP reachable?
# Without this: all files would show as MISSING (SSH timeout per file).
# False alarm from a network hiccup.
#
# check_remote_array() — is the remote /mnt/user mounted?
# Without this: remote files don't exist yet (array not started).
# All files would show as MISSING. False catastrophic alarm.
#
# Both must pass before any checksums are computed.
# ─────────────────────────────────────────────────────────────────────────────
── Configuration ────────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
BACKUP_VERIFY_SAMPLE=10 # random files sampled per share per run
BACKUP_VERIFY_MIN_SIZE="1M" # skip files smaller than this (checksums waste
# time on tiny files with low corruption risk)
# master_host*.conf
# ─────────────────────────────────────────────────────────────────────────────
# Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
# No additional configuration needed for the standard setup.
#
HOST1_BACKUP_VERIFY_SHARES=(
# empty — uses HOST1_DAILY_SYNC_SHARES automatically
# "/mnt/user/Movies" # override here to check specific shares only
)
── Usage ───────────────────────────────────────────────────────────────────
backup_verify.sh # normal run — silent if all samples match
backup_verify.sh --dry-run # show which files would be sampled, no checksums
backup_verify.sh --log # verbose — show checksum comparison per file
backup_verify.sh --status # show share list and sample configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📡 bandwidth_monitor.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
rsync transfer history logging and weekly summary reporting. Called automatically by
rsync.sh after each sync — no manual scheduling needed for the logging part. The
report can be run manually or on a weekly schedule.
# Log mode: called automatically by rsync.sh — no manual scheduling needed
# Report mode — scheduled: 0 11 * * 0 (Sunday 11am weekly)
── Two Modes ────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# --log-transfer profile duration_seconds status bytes_transferred
# Called by rsync.sh after each sync completes.
# Appends one line to BANDWIDTH_LOG.
# Trims entries older than BANDWIDTH_LOG_RETENTION days.
# Flags transfers exceeding BANDWIDTH_WARN_GB for report highlighting.
# You never call this manually.
#
# --report (or no arguments)
# Reads the accumulated log and generates a weekly summary.
# Per-profile breakdown: run count, total transferred, average duration, failures.
# Last 7 days activity timeline.
# Flags any days or transfers that exceeded BANDWIDTH_WARN_GB.
# ─────────────────────────────────────────────────────────────────────────────
── Log Format — Version-Proof Design ──────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# One line per transfer:
# YYYY-MM-DD|HH:MM|profile|duration_seconds|status|bytes_transferred
#
# Example entries:
# 2026-04-14|01:23|critical-data|143|success|2847362048
# 2026-04-14|01:31|arrs_stack|287|success|891234567
# 2026-04-14|02:15|movies|1847|failed|0
#
# Why this format:
# Earlier designs parsed rsync's human-readable output for bytes transferred.
# rsync changes its output format between versions — those parsers break silently.
# Status (success/failed) and duration are always available regardless of rsync version.
# Bytes captured from rsync --stats output via awk — version-stable field names.
# This format survives any rsync update with no changes.
# ─────────────────────────────────────────────────────────────────────────────
── Configuration ────────────────────────────────────────────────────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
BANDWIDTH_LOG="/boot/config/bandwidth_history.db" # survives reboots
BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded, never grows unbounded
BANDWIDTH_WARN_GB=50 # flag transfers or daily totals exceeding this
── Usage ───────────────────────────────────────────────────────────────────
bandwidth_monitor.sh # generate report (default mode)
bandwidth_monitor.sh --report # generate report (explicit)
bandwidth_monitor.sh --status # show config and log statistics
bandwidth_monitor.sh --log # verbose output
# This is called automatically by rsync.sh — never call manually:
bandwidth_monitor.sh --log-transfer "profile" 287 "success" 891234567
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
📋 weekly_health_digest.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Full ecosystem health aggregation. Reads state files from across the entire system — watchdog strikes, failover state, skip list, bandwidth history, transcode stats, cert status — and compiles them into a single digest report. Reads only, writes nothing.
# Scheduled: 0 8 * * * (8am daily — DIGEST_PROFILE controls when it notifies)
── The Key Design — Schedule Daily, Profile Controls Behaviour ─────────────
# master.conf
# ─────────────────────────────────────────────────────────────────────────────
# The cron schedule never changes — always daily at 8am.
# DIGEST_PROFILE controls what actually happens on each run.
# Switch profiles by changing one line in master.conf — no cron edit needed.
#
DIGEST_PROFILE="weekly" # always | smart | weekly
── Three Profiles ───────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# always — sends every run regardless of findings
# Use: daily digest of everything, even when healthy
# Result: daily notification with full system snapshot
#
# smart — sends only when something worth reporting is found
# Use: quiet operation, alerts on issues
# Result: silent on clean days, notification when something needs attention
# What triggers "worth reporting" is configurable via DIGEST_SMART_ON_* toggles
#
# weekly — sends once per week on DIGEST_DAY, silent all other days
# Use: weekly summary without daily noise
# Result: one notification on Sunday, silent Monday-Saturday
# ─────────────────────────────────────────────────────────────────────────────
DIGEST_DAY="Sunday" # for weekly profile — must match `date +%A` output
# Smart profile triggers — each independently toggleable
DIGEST_SMART_ON_WATCHDOG=true # send if any active watchdog strikes
DIGEST_SMART_ON_FAILOVER=true # send if failover state is not NORMAL
DIGEST_SMART_ON_CERT_WARN=true # send if any cert within CERT_WARN_DAYS
DIGEST_SMART_ON_BANDWIDTH=true # send if any transfer exceeded BANDWIDTH_WARN_GB
── Data Sources — Reads Only ────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Every source is a read — this script writes nothing, changes nothing.
#
FAILOVER_STATE_FILE # current failover state (NORMAL/FAILOVER/etc.)
SYS_WATCHDOG_FAILED_FILE # container skip list — anything here needs attention
WATCHDOG_STATE_FILE # active container watchdog strikes
SYS_WATCHDOG_STATE_FILE # active system watchdog strikes
BANDWIDTH_LOG # yesterday's transfer history
TRANSCODE_DAILY_LOG # weekly transcode statistics
CERT_MONITOR_DOMAINS # live openssl cert check per domain
RAMDISK_PATH / TRANSCODE_LINK # current transcode location and usage
# ─────────────────────────────────────────────────────────────────────────────
── Usage ───────────────────────────────────────────────────────────────────
weekly_health_digest.sh # normal run — profile determines if notification sent
weekly_health_digest.sh --dry-run # show digest output, no notification regardless of profile
weekly_health_digest.sh --log # verbose per-section output
weekly_health_digest.sh --status # show profile, day, and trigger configuration
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🎬 emby_session_report.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Emby usage report via the Emby API. Queries activity logs and session history to produce a weekly summary of streaming activity. No persistent state — queries fresh on every run.
# Scheduled: 0 11 * * 0 (Sunday 11am weekly)
── What It Reports ─────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
Server info: name, version, uptime
Active sessions: current streams, direct play vs transcode breakdown
Library stats: movie count, episode count, song count
Activity history: play events from the last EMBY_REPORT_DAYS days
Top content: most played items in the period (top EMBY_REPORT_TOP_N)
Most active users: who watched the most in the period
Transcode ratio: how often transcoding was needed vs direct play
notifies if > 80% of streams are transcoding (config issue)
Ramdisk status: current transcode location and usage
# ─────────────────────────────────────────────────────────────────────────────
── Configuration ────────────────────────────────────────────────────────────
# master_host1.conf
# ─────────────────────────────────────────────────────────────────────────────
# detect_hosts() aliases these from HOST*_ prefixed vars.
# Each server reports on its own Emby instance automatically.
#
HOST1_EMBY_URL="http://192.168.50.2:8096"
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
#
# To get an API key:
# Emby UI → Settings → API Keys → New API Key
# Give it a name (e.g. "unraid_scripts") — copy the key
# master.conf
EMBY_REPORT_DAYS=7 # report period in days
EMBY_REPORT_TOP_N=10 # top N content items to show
── Usage ───────────────────────────────────────────────────────────────────
emby_session_report.sh # generate report
emby_session_report.sh --dry-run # test API connectivity only, no notification
emby_session_report.sh --log # verbose per-section output
emby_session_report.sh --status # show config and Emby connection status
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
⚙️ system_tuning_monitor.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
inotify and PHP-FPM utilisation tracking. Snapshots current usage every 6 hours and writes to a bounded log. The weekly coffee report reads this log for peak/average/ warning counts over the week. Silent when healthy — warns when thresholds are hit.
# Scheduled: 0 */6 * * * (every 6 hours)
── What It Tracks ───────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# inotify instances:
# Current in use vs kernel limit (INOTIFY_MAX_INSTANCES)
# Utilisation % — warns above INOTIFY_WARN_PCT (default 80%)
# Top 5 consumers by instance count
#
# What inotify exhaustion looks like at the application level:
# Downloads complete but arrs don't detect them
# Live TV stutter as media files aren't seen arriving
# Library updates stop happening
# No clear error anywhere — things just stop working
#
# php-fpm workers:
# Active workers vs PHP_MAX_CHILDREN limit
# Utilisation % — warns above PHP_FPM_WARN_PCT (default 80%)
#
# What PHP-FPM exhaustion looks like:
# unRAID WebGUI becomes slow or times out
# Dashboard requests queue behind each other
# Settings saves hang
# ─────────────────────────────────────────────────────────────────────────────
── Log Format ──────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# One line per snapshot — trimmed to TUNING_LOG_RETENTION days on each write:
# DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
# PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
#
# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags — 1 means threshold was exceeded
# during this snapshot. The weekly coffee report counts warnings over the week.
── Configuration ────────────────────────────────────────────────────────────
# master.conf
INOTIFY_WARN_PCT=80 # warn if inotify instances above 80% of limit
PHP_FPM_WARN_PCT=80 # warn if php-fpm workers above 80% of max
TUNING_MONITOR_LOG="$DATA_DIR/tuning_monitor.db"
TUNING_LOG_RETENTION=30 # days — file stays bounded
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🖥️ continuous_scripts_status.sh
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Live dashboard for all continuously running background scripts. Run any time to get an immediate picture of the ecosystem's operational state. No schedule — on demand only.
── What It Shows ────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# For each continuous script (system_watchdog, docker_watchdog, failover):
# Running state, PID, uptime, approximate cycle count
#
# system_watchdog:
# Rootfs usage, free RAM, ZFS ARC size, load average, zombie count, CPU temp
# Active strikes (any check with non-zero strike count)
# Recent watchdog-triggered reboots
#
# docker_watchdog:
# Running vs stopped vs unhealthy container counts
# Required containers — which are running, which are stopped
# Memory-monitored containers — current usage vs hard limits
# Active container strikes
# Recent restart history (per container)
# Skip list — anything here needs human attention
#
# failover:
# Current state (NORMAL/FAILOVER/NO_INTERNET/DARK)
# Which tier containers are active
# Outage duration if in FAILOVER
# Remote server Tailscale visibility (ping result)
# DDNS containers running on this server
# ─────────────────────────────────────────────────────────────────────────────
── Usage ───────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Run any time — no cron, no schedule. This is an on-demand status tool.
# ─────────────────────────────────────────────────────────────────────────────
continuous_scripts_status.sh # live dashboard
continuous_scripts_status.sh --log # verbose output with full state file dumps
━━━ RECOMMENDED SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
# ─────────────────────────────────────────────────────────────────────────────
# Daily
# ─────────────────────────────────────────────────────────────────────────────
0 8 * * * weekly_health_digest.sh # DIGEST_PROFILE controls notify frequency
# ─────────────────────────────────────────────────────────────────────────────
# Every 6 hours — background snapshot
# ─────────────────────────────────────────────────────────────────────────────
0 */6 * * * system_tuning_monitor.sh # inotify + php-fpm utilisation tracking
# ─────────────────────────────────────────────────────────────────────────────
# Sunday morning block — runs after nightly maintenance completes (~3am)
# By 6am the weekly restarts, log clears, and media maintenance have finished.
# Monitors see a freshly maintained system.
# ─────────────────────────────────────────────────────────────────────────────
0 6 * * 0 zfs_memory_snapshot.sh # ZFS + memory — first, before everything
0 7 * * 0 smart_health.sh # drive SMART health
0 9 * * 0 cert_monitor.sh # SSL cert expiry
0 10 * * 0 backup_verify.sh # rsync mirror integrity
0 11 * * 0 emby_session_report.sh # Emby streaming usage
0 11 * * 0 bandwidth_monitor.sh --report # rsync transfer summary
# ─────────────────────────────────────────────────────────────────────────────
# Automatic — no scheduling needed
# ─────────────────────────────────────────────────────────────────────────────
# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each sync.
# continuous_scripts_status.sh is run on demand — no cron entry.
━━━ ADDING A NEW MONITOR ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
The monitor pattern is consistent across all scripts in this folder:
# ─────────────────────────────────────────────────────────────────────────────
# All monitor scripts share these properties:
#
# SILENT_MODE=false — output is the point (monitors are meant to produce output)
# No action taken — observe only, intervention is handled elsewhere
# acquire_lock — prevent concurrent runs producing duplicate output
# detect_hosts() — correct configuration per server via MY_ID
# validate_unraid_cmd — notify script validated before use
# --dry-run support — run checks without sending notifications
# --status support — show configuration and exit
# --log support — verbose per-item output
# Silent on healthy — no notification when everything passes
#
# To add a new monitor:
# 1. Source load_config.sh — gives you all common functions
# 2. Set SILENT_MODE=false — this is a monitor, output is expected
# 3. Call detect_hosts() — correct per-host configuration
# 4. Call acquire_lock — prevent duplicate runs
# 5. Do your checks — read, connect, compare
# 6. notify() only on problems — silent on healthy results
# 7. Support --dry-run (no notify) and --status (show config, exit)
# ─────────────────────────────────────────────────────────────────────────────