FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
bb709b27625d6a6c189a0b44cb2fa524ab4fce96
Varaverk/Monitors/README-Monitors.md
T

13 KiB
Raw Blame History

Monitors

Watch and report only. Scripts in this folder never take action — they observe, measure, and notify. Intervention is handled by other parts of the ecosystem.

unRAID_Essentials/   — acts on the system (restarts, reboots, stops)
Docker_Essentials/   — acts on containers (watchdog, restarts)
Monitors/            — observes and reports (this folder)

All monitor scripts are safe to run at any time. None of them write to flash drives except bandwidth_monitor.sh which makes one bounded append per rsync run. All other monitors are read-only operations.

Scripts

cert_monitor.sh

Monitors SSL certificate expiry for all configured domains.

# Scheduled as: 0 9 * * 0  (Sunday 9am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh

How it works:

Connects directly to each domain via openssl s_client and reads the certificate the server is actually presenting. This is different from checking your certificate files directly — it catches real-world issues that file-based checks miss:

  • Certificate renewed but web server not reloaded — old cert still being served
  • Wrong certificate being served for a domain
  • Certificate chain issues invisible to the cert file itself

Each domain and subdomain is a separate entry. They have independent certificates — gmer4lfe.com and auth.gmer4lfe.com may expire on different dates.

Notification behavior:

  • Silent when all certs are healthy
  • One warning notification for all domains approaching CERT_WARN_DAYS
  • One critical notification for all domains within CERT_CRIT_DAYS
  • Notifications batched by severity — not one per domain

Configuration:

CERT_MONITOR_DOMAINS=(
    "Gmer4Lfe.com"
    "Gmer4Lfe.us"
    # "auth.Gmer4Lfe.com"    # add subdomains as separate entries
)
CERT_WARN_DAYS=30   # warn when this many days remaining
CERT_CRIT_DAYS=7    # critical when this many days remaining
CERT_TIMEOUT=10     # seconds before giving up per domain

smart_health.sh

Checks SMART health attributes for all drives in the system.

# Scheduled as: 0 7 * * 0  (Sunday 7am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh

What it checks per drive:

Attribute Threshold Meaning
Reallocated_Sector_Ct > 0 = warning Bad sectors remapped — drive showing wear
Current_Pending_Sector > 0 = warning Sectors awaiting reallocation
Offline_Uncorrectable > 0 = critical Sectors that could not be corrected
Temperature_Celsius SMART_TEMP_WARN/CRIT Drive running hot
Power_On_Hours informational Drive age estimate
Overall health status PASSED/FAILED Drive's own self-assessment

Drive discovery is automatic — /dev/sd* and /dev/nvme* are scanned on every run. No drive list to maintain.

Why ignore the boot USB:

unRAID boots from a USB flash drive that typically appears as sda. Flash drives either don't support SMART or report meaningless values. Add it to SMART_IGNORE_DRIVES to keep it out of the report.

Notification behavior:

  • Silent when all drives are healthy
  • One warning notification listing all drives with concerning attributes
  • One critical notification if any drive has uncorrectable sectors

Configuration:

SMART_TEMP_WARN=45          # degrees C
SMART_TEMP_CRIT=55          # degrees C
SMART_IGNORE_DRIVES=(
    "sda"                   # boot USB — not meaningful to check
)

zfs_memory_snapshot.sh

Weekly ZFS pool health and memory diagnostic report.

# Scheduled as: 0 6 * * 0  (Sunday 6am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/zfs_memory_snapshot.sh

What it reports:

  • Pool status — ONLINE/DEGRADED/FAULTED per pool
  • Pool overview — size, allocated, free, capacity, health
  • ARC statistics — max, current, metadata usage, utilization %
  • Memory status — total, free, available RAM vs thresholds
  • Top N Docker containers by memory usage
  • Kernel pressure snapshot via vmstat

Informational only. This script reports what it finds. system_watchdog.sh handles threshold-based intervention — ARC reclaim, reboot decisions, memory pressure response. The snapshot gives you the weekly picture; the watchdog handles emergencies.

Output is written to both console and ZFS_REPORT_LOG — the log file lets you compare pool health week over week without having to remember what last week's numbers were.

Pool ignore list:

Pools expected to run at high capacity can be excluded from health reporting. They remain fully monitored by unRAID — this only affects what appears in the weekly report.

ZFS_REPORT_IGNORE_POOLS=(
    "disk10"    # high usage expected
    "disk9"
    "disk8"
    "disk6"
    "disk5"
)

Configuration:

ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log"
ZFS_REPORT_ARC_WARN_PCT=90      # warn if ARC utilization above this %
ZFS_REPORT_FREE_WARN_GB=10      # warn if free RAM below this GB
ZFS_REPORT_AVAIL_WARN_GB=20     # warn if available RAM below this GB
ZFS_REPORT_DOCKER_TOP=10        # top N Docker memory users to show

backup_verify.sh

Verifies the rsync mirror is healthy by comparing random file checksums between local and remote servers.

# Scheduled as: 0 10 * * 0  (Sunday 10am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh

Why this matters:

rsync.sh copies files successfully. But does the copy match the original? backup_verify.sh answers that question by independently computing MD5 checksums on both sides and comparing them. It catches:

  • Silent data corruption during transfer
  • Files that transferred but were corrupted at rest
  • Partial transfers that rsync reported as success
  • Storage hardware issues on either server

How it works:

  1. Randomly samples BACKUP_VERIFY_SAMPLE files per share (files larger than BACKUP_VERIFY_MIN_SIZE)
  2. Computes MD5 checksum locally
  3. SSHes to remote and computes MD5 checksum there
  4. Compares results

Result per file:

Result Meaning
MATCH Checksums identical — file correctly mirrored
MISMATCH File exists on both but checksums differ — sync may have failed
MISSING File exists locally but not on remote — not yet synced or deleted

Configuration:

# Leave empty to use DAILY_SYNC_SHARES automatically
BACKUP_VERIFY_SHARES=(
    # /mnt/user/Movies
    # /mnt/user/Tv_Shows
)
BACKUP_VERIFY_SAMPLE=10     # files sampled per share per run
BACKUP_VERIFY_MIN_SIZE=1M   # skip files smaller than this

If BACKUP_VERIFY_SHARES is empty the script automatically uses DAILY_SYNC_SHARES — no additional configuration needed for the standard setup.

Uses the existing SSH keys already configured for rsync — no additional setup required.

bandwidth_monitor.sh

Logs rsync transfer history and generates weekly summary reports. Called automatically by rsync.sh — you do not need to schedule the logging mode manually.

# Log mode — called automatically by rsync.sh after each successful sync
# No manual scheduling needed

# Report mode — run manually or schedule weekly
# Scheduled as: 0 11 * * 0  (Sunday 11am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/bandwidth_monitor.sh --report

Two modes:

--log-transfer profile duration status — called by rsync.sh after each sync. Appends one line to the log file and trims entries older than BANDWIDTH_LOG_RETENTION days. You never call this manually.

--report — reads the log file and generates a weekly summary showing per-profile run counts, average durations, last 7 days activity, and failure counts.

Log format:

YYYY-MM-DD|HH:MM|profile|duration_seconds|status
2026-04-14|01:23|arrs_stack|287|success
2026-04-14|01:31|critical-data|143|success
2026-04-14|02:15|movies|1847|failed

Why this format:

The log never parses rsync output. Earlier designs tried to extract bytes transferred from rsync's human-readable output — that approach breaks silently when rsync updates and changes its output format. The current format captures what's reliably available: profile, duration, and success/failure. This is version-proof and survives any rsync update.

Flash drive design:

The log lives on /boot/ so it survives reboots. Each rsync run makes exactly one append and one trim — the file never grows beyond BANDWIDTH_LOG_RETENTION lines. Minimal flash wear.

Configuration:

BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90  # days — file stays bounded
BANDWIDTH_WARN_GB=50        # flag days exceeding this in reports
                            # (note: current log tracks duration not bytes)

weekly_health_digest.sh

Aggregates system health data from across the entire ecosystem into a single digest report.

# Scheduled as: 0 8 * * *  (8am daily — profile controls when it notifies)
/mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh

The key design: schedule it daily regardless of profile. The DIGEST_PROFILE setting in Master.conf controls when a notification is actually sent — not the cron schedule.

Three profiles:

Profile Behavior Use When
always Sends every run You want a daily health summary
smart Sends only if something worth reporting Quiet operation, alerts on issues
weekly Sends once per week on DIGEST_DAY Weekly digest, silent other days

Switch profiles by changing DIGEST_PROFILE in Master.conf — no cron changes needed.

Data sources — reads only, no writes:

Source What it reads
/tmp/transcode_state.db Ramdisk symlink and usage
/tmp/container_watchdog_state.db Active container watchdog strikes
/tmp/system_watchdog_state.db Active system watchdog strikes
/boot/config/failover_state.db Current failover state
/boot/config/system_watchdog_failed.db Container skip list
/boot/config/bandwidth_history.db Recent transfer totals
Live openssl connection SSL cert days remaining per domain

Smart profile triggers:

# Set true to include this check in smart mode's "worth reporting" decision
DIGEST_SMART_ON_WATCHDOG=true    # any active watchdog strikes
DIGEST_SMART_ON_FAILOVER=true    # failover state is not NORMAL
DIGEST_SMART_ON_CERT_WARN=true   # any cert under CERT_WARN_DAYS
DIGEST_SMART_ON_BANDWIDTH=true   # any transfer exceeded BANDWIDTH_WARN_GB

Configuration:

DIGEST_PROFILE="weekly"     # always | smart | weekly
DIGEST_DAY="Sunday"         # for weekly profile — must match date +%A output

emby_session_report.sh

Weekly Emby usage report via the Emby API.

# Scheduled as: 0 11 * * 0  (Sunday 11am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/emby_session_report.sh

What it reports:

  • Active streams at time of run
  • Stream breakdown — total, Live TV, transcoding, direct play
  • Library counts — movies, episodes, songs
  • Current ramdisk transcode usage and symlink state

No persistent writes — queries the Emby API fresh on every run. No log files, no state. Run it any time for a current snapshot.

Requires an Emby API key:

  1. Open Emby Settings → API Keys
  2. Generate a new key
  3. Paste it into Master.conf as EMBY_API_KEY

Configuration:

EMBY_URL="http://localhost:8096"
EMBY_API_KEY=""          # get from Emby Settings → API Keys
EMBY_REPORT_DAYS=7       # report period in days
EMBY_REPORT_TOP_N=10     # top N content items to show

Flash Drive Write Policy

unRAID boots from a USB flash drive. Flash drives have limited write cycles. The Monitors folder is designed with this in mind:

Script Writes to flash Notes
cert_monitor.sh Never Read-only openssl checks
smart_health.sh Never Read-only smartctl checks
zfs_memory_snapshot.sh Never Writes to /var/log/ (RAM disk)
backup_verify.sh Never SSH + MD5 comparison only
bandwidth_monitor.sh One append + one trim per rsync run Bounded — never exceeds retention days
weekly_health_digest.sh Never Reads existing state files only
emby_session_report.sh Never API queries only

The only flash write in the entire Monitors folder is bandwidth_monitor.sh — and it's designed to be minimal and bounded.

Recommended Schedule

# Daily
0 8 * * *    weekly_health_digest.sh     # profile controls when it notifies

# Weekly — Sunday morning block
0 6 * * 0    zfs_memory_snapshot.sh
0 7 * * 0    smart_health.sh
0 9 * * 0    cert_monitor.sh
0 10 * * 0   backup_verify.sh
0 11 * * 0   emby_session_report.sh
0 11 * * 0   bandwidth_monitor.sh --report

# Automatic — no scheduling needed
# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each sync

The Sunday morning block runs after the nightly maintenance window — by the time the monitors run, the weekly restarts, log clears, and media management jobs have completed. The health snapshot reflects a freshly maintained system.

--dry-run Support

All monitor scripts support --dry-run. In dry-run mode:

  • Checks run and results are shown
  • No notifications are sent
  • No files are written

Useful for testing configuration changes before scheduling:

/mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --dry-run
/mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh --dry-run
/mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh --dry-run
Reference in New Issue View Git Blame Copy Permalink