feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,532 @@
|
||||
# ━━━━━ MONITORS — Manual ━━━━━
|
||||
|
||||
Config reference, procedures, operational workflows.
|
||||
For overview see README-Monitors.md. For per-script detail see script headers.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CERT MONITOR ━━━
|
||||
|
||||
### Domain Configuration
|
||||
|
||||
Each domain and subdomain has an independent TLS certificate and must be listed
|
||||
separately. `cert_monitor.sh` makes one openssl connection per entry.
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_CERT_MONITOR_DOMAINS=(
|
||||
"Gmer4Lfe.com"
|
||||
"Gmer4Lfe.us"
|
||||
# "auth.Gmer4Lfe.com" # add subdomains separately
|
||||
# "cloud.Gmer4Lfe.com"
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
HOST2_CERT_MONITOR_DOMAINS=(
|
||||
"Jayred365.com"
|
||||
# "auth.Jayred365.com"
|
||||
)
|
||||
```
|
||||
|
||||
Each server monitors its own domains. `detect_hosts()` aliases the correct list
|
||||
to `CERT_MONITOR_DOMAINS` based on `MY_ID`.
|
||||
|
||||
### Threshold Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
CERT_WARN_DAYS=30 # warn this many days before expiry — time to investigate
|
||||
CERT_CRIT_DAYS=7 # critical alert — action needed now
|
||||
CERT_TIMEOUT=10 # seconds per domain before declaring FAILED
|
||||
```
|
||||
|
||||
**Sizing guidance:**
|
||||
- `CERT_WARN_DAYS=30` gives a month to investigate and renew. If Let's Encrypt
|
||||
auto-renewal is working, you'll only see warnings when renewal breaks.
|
||||
- `CERT_CRIT_DAYS=7` — at 7 days remaining, manual action is needed today.
|
||||
- `CERT_TIMEOUT=10` — sufficient for typical public domains; increase for slow
|
||||
DNS resolution or high-latency connections.
|
||||
|
||||
### Notification Behavior
|
||||
|
||||
| State | Condition | Action |
|
||||
|-------|-----------|--------|
|
||||
| HEALTHY | > CERT_WARN_DAYS remaining | Silent — no notification |
|
||||
| WARNING | ≤ CERT_WARN_DAYS remaining | One notification listing all WARNING domains |
|
||||
| CRITICAL | ≤ CERT_CRIT_DAYS remaining | Separate notification listing all CRITICAL domains |
|
||||
| FAILED | Could not connect or parse cert | Notification — treat as critical |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SMART HEALTH ━━━
|
||||
|
||||
### Drive Ignore List
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_SMART_IGNORE_DRIVES=(
|
||||
"sda" # boot USB flash drive — no meaningful SMART data
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
HOST2_SMART_IGNORE_DRIVES=(
|
||||
"sda" # boot USB flash drive
|
||||
)
|
||||
```
|
||||
|
||||
unRAID typically boots from a USB flash drive that appears as `sda`. Flash drives
|
||||
either don't support SMART or report values that are meaningless for health assessment.
|
||||
Add any drive that produces spurious SMART data here.
|
||||
|
||||
### Temperature Configuration
|
||||
|
||||
```bash
|
||||
# master.conf — fallback values if dynamix.cfg is not found
|
||||
SMART_TEMP_WARN=45 # °C
|
||||
SMART_TEMP_CRIT=55 # °C
|
||||
```
|
||||
|
||||
`smart_health.sh` reads temperature thresholds directly from
|
||||
`/boot/config/plugins/dynamix/dynamix.cfg` at runtime, using the same hot/max/hotssd/maxssd
|
||||
values as unRAID's dashboard. The `master.conf` values are only used as a fallback
|
||||
if `dynamix.cfg` is not found (e.g., running outside of unRAID).
|
||||
|
||||
### Attribute Reference
|
||||
|
||||
| Attribute | Threshold | Meaning |
|
||||
|-----------|-----------|---------|
|
||||
| `Reallocated_Sector_Ct` | Any > 0 is warning | Drive found bad sectors and swapped in spares. 0 = healthy. Count growing = degrading. |
|
||||
| `Current_Pending_Sector` | Any > 0 is warning | Sectors suspected bad, not yet confirmed. May recover on next read, may escalate. Watch it. |
|
||||
| `Offline_Uncorrectable` | Any > 0 is critical | Could not correct during offline tests. No spares. Data loss risk. |
|
||||
| Overall SMART status | FAILED = immediate alert | Drive's own self-assessment. FAILED = get data off now. |
|
||||
| `Temperature_Celsius` | vs dynamix.cfg thresholds | Sustained high temp shortens drive life significantly. |
|
||||
| `Power_On_Hours` | Informational | Drive age in days — useful context for other attributes. |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ BACKUP VERIFY ━━━
|
||||
|
||||
### Share Configuration
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_BACKUP_VERIFY_SHARES=(
|
||||
# empty — uses HOST1_DAILY_SYNC_SHARES automatically
|
||||
# "/mnt/user/Movies" # override to check specific shares only
|
||||
)
|
||||
```
|
||||
|
||||
Leave `HOST*_BACKUP_VERIFY_SHARES` empty and `backup_verify.sh` will automatically
|
||||
verify the same shares configured for daily rsync. Only populate it if you want to
|
||||
verify a different set.
|
||||
|
||||
### Sample Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
BACKUP_VERIFY_SAMPLE=10 # random files sampled per share per run
|
||||
BACKUP_VERIFY_MIN_SIZE="1M" # skip files smaller than this
|
||||
```
|
||||
|
||||
**Sizing guidance:**
|
||||
- `BACKUP_VERIFY_SAMPLE=10` is a spot check — catches systematic hardware problems
|
||||
while running in minutes. Increase to 25-50 for deeper confidence on a large library;
|
||||
decrease to 5 if the run is too slow against many shares.
|
||||
- `BACKUP_VERIFY_MIN_SIZE="1M"` — tiny files (NFO, thumbnails, subtitles) have negligible
|
||||
corruption risk and each one adds an SSH round-trip. Skip them.
|
||||
|
||||
### Result States
|
||||
|
||||
| State | Meaning | Action |
|
||||
|-------|---------|--------|
|
||||
| MATCH | Checksums identical on both servers | Correctly mirrored — no action |
|
||||
| MISMATCH | File exists on both, checksums differ | Sync failure or corruption — investigate |
|
||||
| MISSING | File exists locally, not on remote | Not yet synced or deleted on remote — may be normal |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ BANDWIDTH MONITOR ━━━
|
||||
|
||||
### Log Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
BANDWIDTH_LOG="/boot/config/bandwidth_history.db" # survives reboots
|
||||
BANDWIDTH_LOG_RETENTION=90 # days — file stays bounded, never grows unbounded
|
||||
BANDWIDTH_WARN_GB=50 # flag transfers or daily totals exceeding this
|
||||
```
|
||||
|
||||
**Why `/boot/config/`**: The log needs to survive reboots to build a useful history.
|
||||
`/boot/config/` is on the USB flash drive, which survives reboots and is backed up
|
||||
by unRAID's flash backup. The log is bounded by `BANDWIDTH_LOG_RETENTION` so it never
|
||||
grows unbounded.
|
||||
|
||||
**`BANDWIDTH_WARN_GB`**: Set to a value that represents "unexpectedly large" for your
|
||||
setup. If your typical daily sync transfers 5-10GB, 50GB would flag a full-library resync.
|
||||
Adjust based on your share sizes.
|
||||
|
||||
### Log Format
|
||||
|
||||
One line per transfer — do not edit manually:
|
||||
```
|
||||
YYYY-MM-DD|HH:MM|profile|duration_seconds|status|bytes_transferred
|
||||
|
||||
# Example entries:
|
||||
2026-04-14|01:23|critical-data|143|success|2847362048
|
||||
2026-04-14|01:31|arrs_stack|287|success|891234567
|
||||
2026-04-14|02:15|movies|1847|failed|0
|
||||
```
|
||||
|
||||
### How rsync.sh Calls bandwidth_monitor.sh
|
||||
|
||||
`rsync.sh` automatically calls `bandwidth_monitor.sh --log-transfer` after each sync
|
||||
profile completes. You never need to call `--log-transfer` manually. The call:
|
||||
|
||||
```bash
|
||||
bandwidth_monitor.sh --log-transfer "movies" 1847 "failed" 0
|
||||
# profile secs status bytes
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WEEKLY HEALTH DIGEST ━━━
|
||||
|
||||
### Profile Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
DIGEST_PROFILE="weekly" # always | smart | weekly
|
||||
DIGEST_DAY="Sunday" # for weekly profile — must match `date +%A` output
|
||||
```
|
||||
|
||||
The cron schedule is always `0 8 * * *` (8am daily). Change only `DIGEST_PROFILE`
|
||||
to switch behavior — no cron edit needed.
|
||||
|
||||
**Profile guide:**
|
||||
|
||||
| Profile | When to use | Result |
|
||||
|---------|-------------|--------|
|
||||
| `always` | You want a daily check-in regardless of system state | Notification every morning |
|
||||
| `smart` | Quiet operation, only alert on real issues | Silent when healthy, notification when something needs attention |
|
||||
| `weekly` | One weekly summary is enough | Single notification on `DIGEST_DAY`, silent all other days |
|
||||
|
||||
### Smart Profile Triggers
|
||||
|
||||
```bash
|
||||
# master.conf — each independently toggleable
|
||||
DIGEST_SMART_ON_WATCHDOG=true # send if any active watchdog strikes
|
||||
DIGEST_SMART_ON_FALLBACK=true # send if fallback state is not NORMAL
|
||||
DIGEST_SMART_ON_CERT_WARN=true # send if any cert within CERT_WARN_DAYS
|
||||
DIGEST_SMART_ON_BANDWIDTH=true # send if any transfer exceeded BANDWIDTH_WARN_GB
|
||||
```
|
||||
|
||||
Turn any trigger off by setting it to `false`. The digest still runs and collects
|
||||
data — it just doesn't trigger a notification for that condition.
|
||||
|
||||
### Data Sources
|
||||
|
||||
`weekly_health_digest.sh` reads (but never writes):
|
||||
|
||||
| State File | Source | What It Shows |
|
||||
|-----------|--------|---------------|
|
||||
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `Docker_Essentials/docker_watchdog.sh` | Container skip list — needs human attention |
|
||||
| `WATCHDOG_STATE_FILE` | `Docker_Essentials/docker_watchdog.sh` | Active container strike counts |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `unRAID_Essentials/system_watchdog.sh` | Active system watchdog strikes |
|
||||
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
|
||||
| `TRANSCODE_DAILY_LOG` | `Transcodes/` | Weekly transcode statistics |
|
||||
| `CERT_MONITOR_DOMAINS` | live openssl check | Current cert status per domain |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ EMBY SESSION REPORT ━━━
|
||||
|
||||
### Emby Credentials
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
||||
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
|
||||
|
||||
# master_host2.conf
|
||||
HOST2_EMBY_URL="http://192.168.50.3:8096"
|
||||
HOST2_EMBY_API_KEY="<host2_api_key>"
|
||||
```
|
||||
|
||||
To generate an API key: Emby UI → Settings → API Keys → New API Key.
|
||||
Give it a descriptive name (e.g., `unraid_scripts`). The key is only shown once.
|
||||
|
||||
### Report Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
EMBY_REPORT_DAYS=7 # report period in days
|
||||
EMBY_REPORT_TOP_N=10 # top N content items to show
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SYSTEM TUNING MONITOR ━━━
|
||||
|
||||
### Threshold Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
INOTIFY_WARN_PCT=80 # warn if inotify instances above 80% of kernel limit
|
||||
PHP_FPM_WARN_PCT=80 # warn if php-fpm workers above 80% of max
|
||||
TUNING_MONITOR_LOG="$DATA_DIR/tuning_monitor.db"
|
||||
TUNING_LOG_RETENTION=30 # days — log stays bounded
|
||||
```
|
||||
|
||||
**`INOTIFY_WARN_PCT`**: 80% leaves headroom before kernel-level exhaustion. If you
|
||||
regularly see warnings at 80% and your system is stable, increase to 90%. Don't set
|
||||
higher than 90% — at 100% utilisation new inotify watches silently fail.
|
||||
|
||||
**`PHP_FPM_WARN_PCT`**: 80% means the WebGUI is using most of its workers. At 100%
|
||||
new requests queue (WebGUI feels sluggish) or time out.
|
||||
|
||||
**`PHP_MAX_CHILDREN`**: Set by `php_fpm_max_children.sh` in `unRAID_Essentials/` — do
|
||||
not set manually here.
|
||||
|
||||
### Reading the Weekly Digest Data
|
||||
|
||||
`system_tuning_monitor.sh` feeds data into `weekly_health_digest.sh`. The digest
|
||||
shows for the week:
|
||||
- inotify: peak utilisation, average utilisation, number of snapshots that hit the warning threshold
|
||||
- php-fpm: same
|
||||
|
||||
A few warning snapshots per week is normal. A rising peak or many warnings per week
|
||||
means the limits should be adjusted — use `inotify_tuning.sh` or `php_fpm_max_children.sh`
|
||||
in `unRAID_Essentials/`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ ZFS MEMORY SNAPSHOT ━━━
|
||||
|
||||
### Pool Ignore List
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk10" # JBOD member — high usage expected, exclude from report noise
|
||||
"disk9"
|
||||
"disk8"
|
||||
"disk6"
|
||||
"disk5"
|
||||
)
|
||||
|
||||
# master_host2.conf
|
||||
HOST2_ZFS_REPORT_IGNORE_POOLS=(
|
||||
# list host2's JBOD members here
|
||||
)
|
||||
```
|
||||
|
||||
In a JBOD ZFS configuration each disk appears as its own pool. These single-disk
|
||||
pools normally show high capacity utilisation by design — including them would produce
|
||||
constant warnings. The ignore list removes them from the report while unRAID's built-in
|
||||
monitoring continues to watch them.
|
||||
|
||||
### Threshold Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log" # tmpfs — resets on reboot
|
||||
ZFS_REPORT_ARC_WARN_PCT=90 # warn if ARC using more than 90% of its max
|
||||
ZFS_REPORT_FREE_WARN_GB=10 # warn if free RAM below 10GB
|
||||
ZFS_REPORT_AVAIL_WARN_GB=20 # warn if available RAM below 20GB
|
||||
ZFS_REPORT_DOCKER_TOP=10 # top 10 Docker containers by memory
|
||||
```
|
||||
|
||||
**`ZFS_REPORT_LOG`**: Written to `/var/log/` (tmpfs) — resets on reboot. This is
|
||||
intentional — the log is for the current uptime's week-over-week comparison, not
|
||||
long-term history. It avoids flash drive writes entirely.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FULL CONFIGURATION REFERENCE ━━━
|
||||
|
||||
### master.conf
|
||||
|
||||
```bash
|
||||
# cert_monitor.sh
|
||||
CERT_WARN_DAYS=30
|
||||
CERT_CRIT_DAYS=7
|
||||
CERT_TIMEOUT=10
|
||||
|
||||
# smart_health.sh
|
||||
SMART_TEMP_WARN=45
|
||||
SMART_TEMP_CRIT=55
|
||||
|
||||
# backup_verify.sh
|
||||
BACKUP_VERIFY_SAMPLE=10
|
||||
BACKUP_VERIFY_MIN_SIZE="1M"
|
||||
|
||||
# bandwidth_monitor.sh
|
||||
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
|
||||
BANDWIDTH_LOG_RETENTION=90
|
||||
BANDWIDTH_WARN_GB=50
|
||||
|
||||
# weekly_health_digest.sh
|
||||
DIGEST_PROFILE="weekly"
|
||||
DIGEST_DAY="Sunday"
|
||||
DIGEST_SMART_ON_WATCHDOG=true
|
||||
DIGEST_SMART_ON_FALLBACK=true
|
||||
DIGEST_SMART_ON_CERT_WARN=true
|
||||
DIGEST_SMART_ON_BANDWIDTH=true
|
||||
|
||||
# emby_session_report.sh
|
||||
EMBY_REPORT_DAYS=7
|
||||
EMBY_REPORT_TOP_N=10
|
||||
|
||||
# system_tuning_monitor.sh
|
||||
INOTIFY_WARN_PCT=80
|
||||
PHP_FPM_WARN_PCT=80
|
||||
TUNING_MONITOR_LOG="$DATA_DIR/tuning_monitor.db"
|
||||
TUNING_LOG_RETENTION=30
|
||||
|
||||
# zfs_memory_snapshot.sh
|
||||
ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log"
|
||||
ZFS_REPORT_ARC_WARN_PCT=90
|
||||
ZFS_REPORT_FREE_WARN_GB=10
|
||||
ZFS_REPORT_AVAIL_WARN_GB=20
|
||||
ZFS_REPORT_DOCKER_TOP=10
|
||||
```
|
||||
|
||||
### master_host*.conf
|
||||
|
||||
```bash
|
||||
# cert_monitor.sh
|
||||
HOST1_CERT_MONITOR_DOMAINS=("domain1.com" "domain2.com")
|
||||
HOST2_CERT_MONITOR_DOMAINS=("domain3.com")
|
||||
|
||||
# smart_health.sh
|
||||
HOST1_SMART_IGNORE_DRIVES=("sda")
|
||||
HOST2_SMART_IGNORE_DRIVES=("sda")
|
||||
|
||||
# backup_verify.sh (leave empty to use DAILY_SYNC_SHARES)
|
||||
HOST1_BACKUP_VERIFY_SHARES=()
|
||||
HOST2_BACKUP_VERIFY_SHARES=()
|
||||
|
||||
# emby_session_report.sh
|
||||
HOST1_EMBY_URL="http://192.168.50.2:8096"
|
||||
HOST1_EMBY_API_KEY="<key>"
|
||||
HOST2_EMBY_URL="http://192.168.50.3:8096"
|
||||
HOST2_EMBY_API_KEY="<key>"
|
||||
|
||||
# zfs_memory_snapshot.sh
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=("disk10" "disk9" "disk8" "disk6" "disk5")
|
||||
HOST2_ZFS_REPORT_IGNORE_POOLS=()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RECOMMENDED SCHEDULE ━━━
|
||||
|
||||
```bash
|
||||
# Daily
|
||||
0 8 * * * weekly_health_digest.sh # DIGEST_PROFILE controls notify frequency
|
||||
|
||||
# Every 6 hours — background snapshot
|
||||
0 */6 * * * system_tuning_monitor.sh # inotify + php-fpm utilisation tracking
|
||||
|
||||
# Sunday morning block — runs after nightly maintenance completes (~3am)
|
||||
# By 6am the weekly restarts, log clears, and media maintenance have finished.
|
||||
# Monitors see a freshly maintained system.
|
||||
0 6 * * 0 zfs_memory_snapshot.sh # ZFS + memory — first, before everything
|
||||
0 7 * * 0 smart_health.sh # drive SMART health
|
||||
0 9 * * 0 cert_monitor.sh # SSL cert expiry
|
||||
0 10 * * 0 backup_verify.sh # rsync mirror integrity
|
||||
0 11 * * 0 emby_session_report.sh # Emby streaming usage
|
||||
0 11 * * 0 bandwidth_monitor.sh --report # rsync transfer summary
|
||||
|
||||
# Automatic — no scheduling needed
|
||||
# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each sync.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ FLAG REFERENCE ━━━
|
||||
|
||||
All monitor scripts support the same four flags:
|
||||
|
||||
### --dry-run
|
||||
|
||||
Runs checks without sending any notifications. Useful for:
|
||||
- First-time setup validation (verify the script finds your domains/drives/shares)
|
||||
- Ad-hoc health checks without triggering notification noise
|
||||
- Testing after configuration changes
|
||||
|
||||
```bash
|
||||
cert_monitor.sh --dry-run # check all certs, show results, no notify
|
||||
smart_health.sh --dry-run # show which drives would be checked
|
||||
backup_verify.sh --dry-run # show which files would be sampled
|
||||
bandwidth_monitor.sh --dry-run # generate report without sending
|
||||
weekly_health_digest.sh --dry-run # generate digest, no notify regardless of profile
|
||||
zfs_memory_snapshot.sh --dry-run # console output only, no log write
|
||||
system_tuning_monitor.sh --dry-run # measure and show, no log write
|
||||
emby_session_report.sh --dry-run # test connectivity, generate report, no notify
|
||||
```
|
||||
|
||||
### --status
|
||||
|
||||
Shows current configuration and exits without running checks. Use to verify
|
||||
configuration is loaded correctly after editing `master.conf` or `master_host*.conf`.
|
||||
|
||||
```bash
|
||||
cert_monitor.sh --status # domain list, CERT_WARN_DAYS, CERT_CRIT_DAYS, timeout
|
||||
smart_health.sh --status # ignore list, temperature thresholds
|
||||
backup_verify.sh --status # share list, sample size, min file size
|
||||
bandwidth_monitor.sh --status # log path, retention, warn threshold, log stats
|
||||
weekly_health_digest.sh --status # profile, DIGEST_DAY, smart trigger settings
|
||||
zfs_memory_snapshot.sh --status # pool ignore list, memory thresholds
|
||||
system_tuning_monitor.sh --status # warn thresholds, log path, retention
|
||||
emby_session_report.sh --status # Emby URL, API key (masked), report period
|
||||
```
|
||||
|
||||
### --log
|
||||
|
||||
Verbose per-item output. Every domain/drive/file/section logs its result explicitly
|
||||
instead of being silent on pass. Use when investigating or after a configuration change.
|
||||
|
||||
### Normal run (no flags)
|
||||
|
||||
Silent on healthy — only problems produce output or notifications.
|
||||
`weekly_health_digest.sh` is the exception: it always generates a digest, but
|
||||
`DIGEST_PROFILE` controls whether a notification is sent.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ ADDING A NEW MONITOR ━━━
|
||||
|
||||
All monitor scripts share the same pattern:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# [standard header with PURPOSE / SAFEGUARDS / CONFIGURATION / RUNTIME MODES]
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
SILENT_MODE=false # monitor script — output is the point
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# root check if needed
|
||||
# tool validation (validate_unraid_cmd)
|
||||
acquire_lock
|
||||
detect_hosts # if host-specific config needed
|
||||
|
||||
# [status block]
|
||||
# [dry-run block]
|
||||
|
||||
# do checks
|
||||
# notify() only on problems — silent on healthy
|
||||
```
|
||||
|
||||
Key properties:
|
||||
- `SILENT_MODE=false` — monitors produce output
|
||||
- `acquire_lock` — prevent duplicate runs
|
||||
- `notify()` only on problems — never notify on healthy results
|
||||
- `--dry-run` skips `notify()` calls entirely
|
||||
- `--status` shows config and exits before any checks run
|
||||
- `--log` enables verbose per-item output
|
||||
+95
-921
File diff suppressed because it is too large
Load Diff
+90
-36
@@ -1,49 +1,103 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Backup Verify ==============================================
|
||||
# ============================= Backup Verify ==================================================
|
||||
# ==============================================================================================
|
||||
# Verifies the rsync mirror is healthy by comparing random file samples between
|
||||
# local and remote servers using MD5 checksums.
|
||||
#
|
||||
# ── WHAT IT DOES ──────────────────────────────────────────────────────────────────────────────
|
||||
# Randomly samples BACKUP_VERIFY_SAMPLE files per share above BACKUP_VERIFY_MIN_SIZE,
|
||||
# computes MD5 checksums locally, then computes the same checksums on the remote via SSH
|
||||
# and compares results. Catches silent corruption or incomplete syncs that rsync itself
|
||||
# would not detect.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# rsync mirror integrity verification via independent MD5 checksums. Scheduled
|
||||
# weekly (Sunday 10am). Randomly samples BACKUP_VERIFY_SAMPLE files per share
|
||||
# above BACKUP_VERIFY_MIN_SIZE, computes checksums locally, then computes the
|
||||
# same checksums on the remote via SSH and compares.
|
||||
#
|
||||
# ── RESULTS PER FILE ──────────────────────────────────────────────────────────────────────────
|
||||
# MATCH — checksums identical, file is correctly mirrored ✅
|
||||
# MISMATCH — file exists on both but checksums differ — sync may have partially failed
|
||||
# MISSING — file exists locally but not on remote — not yet synced or deleted on remote
|
||||
# Per file: MATCH (checksums identical) | MISMATCH (file exists on both but
|
||||
# checksums differ — sync failure or corruption) | MISSING (file exists locally
|
||||
# but not on remote). All MISMATCHes and significant MISSINGs trigger notification.
|
||||
# rsync exit code 0 is not trusted — this script verifies actual content.
|
||||
#
|
||||
# ── SHARE SELECTION ───────────────────────────────────────────────────────────────────────────
|
||||
# Uses HOST*_BACKUP_VERIFY_SHARES if defined, falls back to HOST*_DAILY_SYNC_SHARES.
|
||||
# Both aliased by detect_hosts() — no manual HOST1/HOST2 selection needed.
|
||||
# Share list from HOST*_BACKUP_VERIFY_SHARES if defined, otherwise falls back
|
||||
# to HOST*_DAILY_SYNC_SHARES. Both aliased by detect_hosts().
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent runs producing duplicate/conflicting results
|
||||
# check_connectivity() — verifies remote reachable before attempting SSH calls
|
||||
# check_remote_array() — verifies remote array mounted before checksums
|
||||
# remote array down = all files "missing" = false alarm ✅
|
||||
# version parity — verifies both servers on compatible unRAID before trusting results
|
||||
# SSH_TIMEOUT — all SSH calls protected against hangs
|
||||
# validate_unraid_cmd — notify script validated before use
|
||||
# Silent by default — only issues produce output, all-match runs are silent
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_BACKUP_VERIFY_SHARES — override share list (empty = use DAILY_SYNC_SHARES)
|
||||
# HOST*_DAILY_SYNC_SHARES — fallback share list
|
||||
# All aliased by detect_hosts()
|
||||
# Independent Verification
|
||||
# rsync reports success when the transfer completed without network errors and
|
||||
# file sizes and modification times match. It does not detect silent corruption
|
||||
# during transfer (bitflip in transit), corruption written to storage at rest
|
||||
# (faulty drive sector), or files that matched size/mtime but had wrong content.
|
||||
# All of these produce exit code 0. This script checks whether "done" means "correct."
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# BACKUP_VERIFY_SAMPLE — random files to check per share (default 10)
|
||||
# BACKUP_VERIFY_MIN_SIZE — minimum file size to include in sample (default 1M)
|
||||
# Intentionally Small Sample
|
||||
# 10 files per share (default) — a spot check, not an exhaustive verify.
|
||||
# Catches systematic problems and hardware issues while running in minutes, not
|
||||
# hours. Full verification would take longer than the rsync itself.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs producing conflicting results.
|
||||
#
|
||||
# Remote Connectivity Check
|
||||
# check_connectivity() verifies the remote Tailscale IP is reachable before
|
||||
# any SSH calls. Without this, all files show as MISSING on a network hiccup.
|
||||
#
|
||||
# Remote Array Check
|
||||
# check_remote_array() verifies /mnt/user is mounted on the remote before
|
||||
# computing checksums. Array not started = all files "missing" = false alarm.
|
||||
#
|
||||
# Version Parity
|
||||
# Refuses to run if remote unRAID version doesn't match local. A mismatch
|
||||
# may mean the remote is in an unexpected state.
|
||||
#
|
||||
# SSH Timeout
|
||||
# SSH_TIMEOUT caps all SSH calls. One hung connection does not block the run.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_BACKUP_VERIFY_SHARES
|
||||
# Shares to verify. Leave empty to use HOST*_DAILY_SYNC_SHARES automatically.
|
||||
# Aliased by detect_hosts() → BACKUP_VERIFY_SHARES.
|
||||
#
|
||||
# HOST*_DAILY_SYNC_SHARES
|
||||
# Fallback share list if BACKUP_VERIFY_SHARES is empty. Aliased by detect_hosts().
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# BACKUP_VERIFY_SAMPLE
|
||||
# Random files checked per share per run. (default: 10)
|
||||
#
|
||||
# BACKUP_VERIFY_MIN_SIZE
|
||||
# Minimum file size to include in sample — tiny files have low corruption
|
||||
# risk and slow checksums. (default: 1M)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# backup_verify.sh
|
||||
# Sample files from all shares and compare checksums. Notify on MISMATCH
|
||||
# or significant MISSING count. Silent when all samples match.
|
||||
#
|
||||
# backup_verify.sh --dry-run
|
||||
# Show which files would be sampled. No checksums computed, no notifications.
|
||||
#
|
||||
# backup_verify.sh --status
|
||||
# Show share list, sample size, and min file size configuration. Then exit.
|
||||
#
|
||||
# backup_verify.sh --log
|
||||
# Verbose per-file checksum comparison output during the run.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# backup_verify.sh — normal run
|
||||
# backup_verify.sh --dry-run — show sample selection only, no checksums
|
||||
# backup_verify.sh --log — verbose output
|
||||
# backup_verify.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -1,46 +1,89 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Bandwidth Monitor ==========================================
|
||||
# ============================= Bandwidth Monitor ==============================================
|
||||
# ==============================================================================================
|
||||
# Logs rsync transfer history and generates weekly summary reports.
|
||||
# Designed for minimal flash drive impact — one bounded write per rsync run.
|
||||
#
|
||||
# ── TWO MODES ─────────────────────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# rsync transfer history logging and weekly summary reporting. Log mode is called
|
||||
# automatically by rsync.sh after each sync — no manual scheduling needed for
|
||||
# logging. Report mode scheduled weekly (Sunday 11am).
|
||||
#
|
||||
# --log-transfer "profile" duration_seconds status
|
||||
# Called automatically by rsync.sh after each sync completes.
|
||||
# Appends one line to the log and trims entries older than BANDWIDTH_LOG_RETENTION.
|
||||
# Flags syncs exceeding BANDWIDTH_WARN_GB in the log for weekly report highlighting.
|
||||
# Log mode (--log-transfer): appends one line per sync to BANDWIDTH_LOG, trims
|
||||
# entries older than BANDWIDTH_LOG_RETENTION days. One bounded write per rsync run.
|
||||
#
|
||||
# --report (or no args)
|
||||
# Generates a summary from the accumulated log.
|
||||
# Shows per-profile breakdown, last 7 days, and overall totals.
|
||||
# This is a monitor script — SILENT_MODE=false — output is the point.
|
||||
# Report mode (default): reads the accumulated log and generates a summary —
|
||||
# per-profile breakdown, run count, total transferred, average duration, failures,
|
||||
# last 7 days activity timeline, any transfers or days exceeding BANDWIDTH_WARN_GB.
|
||||
#
|
||||
# ── LOG FORMAT ────────────────────────────────────────────────────────────────────────────────
|
||||
# One line per transfer — version-proof, never needs rsync output parsing:
|
||||
# YYYY-MM-DD|HH:MM|profile|duration_seconds|status|bytes_transferred
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Log file stays bounded to BANDWIDTH_LOG_RETENTION days — trimmed on every write.
|
||||
# Minimal flash drive impact: one append + one trim per rsync run.
|
||||
# Version-Proof Log Format
|
||||
# Earlier designs parsed rsync's human-readable output for bytes transferred.
|
||||
# rsync changes its output format between versions — those parsers break silently.
|
||||
# The log line (YYYY-MM-DD|HH:MM|profile|duration|status|bytes) captures bytes
|
||||
# from rsync --stats via awk using version-stable field names. Survives any rsync
|
||||
# update with no changes.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock "wait" — prevents log corruption from concurrent rsync completions
|
||||
# validate_unraid_cmd — notify script validated before use
|
||||
# Atomic log write — temp file + mv prevents partial writes on trim
|
||||
# Log existence check — creates log directory if needed, exits cleanly if unwritable
|
||||
# Minimal Flash Drive Impact
|
||||
# unRAID boots from USB flash. One append + one trim per rsync run. The log file
|
||||
# is bounded to BANDWIDTH_LOG_RETENTION days and never grows unbounded.
|
||||
# Atomic write: tmp file + mv prevents partial writes during trim.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# BANDWIDTH_LOG — log file path
|
||||
# BANDWIDTH_LOG_RETENTION — days before old entries are purged (default 90)
|
||||
# BANDWIDTH_WARN_GB — flag syncs larger than this in report (default 50)
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock With Wait
|
||||
# acquire_lock "wait" — multiple rsync profiles may complete close together.
|
||||
# Waiting (not skipping) prevents log entries from being lost.
|
||||
#
|
||||
# Atomic Log Write
|
||||
# Trim uses tmp file + mv — partial writes on log trim cannot corrupt the log.
|
||||
#
|
||||
# Log Directory Guard
|
||||
# Creates the log directory if it doesn't exist. Exits cleanly if unwritable.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# BANDWIDTH_LOG
|
||||
# Log file path. Stored on /boot/ to survive reboots.
|
||||
# (default: /boot/config/bandwidth_history.db)
|
||||
#
|
||||
# BANDWIDTH_LOG_RETENTION
|
||||
# Days before old entries are purged. File stays bounded. (default: 90)
|
||||
#
|
||||
# BANDWIDTH_WARN_GB
|
||||
# Flag transfers or daily totals exceeding this in the report. (default: 50)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# bandwidth_monitor.sh
|
||||
# Generate transfer history report from accumulated log.
|
||||
#
|
||||
# bandwidth_monitor.sh --report
|
||||
# Generate report (explicit form).
|
||||
#
|
||||
# bandwidth_monitor.sh --log-transfer profile secs status bytes
|
||||
# Log a completed rsync transfer. Called by rsync.sh — do not call manually.
|
||||
#
|
||||
# bandwidth_monitor.sh --status
|
||||
# Show log path, retention, warn threshold, and log statistics. Then exit.
|
||||
#
|
||||
# bandwidth_monitor.sh --log
|
||||
# Verbose output during report generation.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# bandwidth_monitor.sh — generate report
|
||||
# bandwidth_monitor.sh --report — generate report (explicit)
|
||||
# bandwidth_monitor.sh --log-transfer profile secs ok — log a transfer (called by rsync.sh)
|
||||
# bandwidth_monitor.sh --status — show config and exit
|
||||
# bandwidth_monitor.sh --log — verbose output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
+76
-41
@@ -1,55 +1,90 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Certificate Monitor ========================================
|
||||
# ============================= Certificate Monitor ============================================
|
||||
# ==============================================================================================
|
||||
# Monitors SSL certificate expiry for all configured domains by connecting directly
|
||||
# via openssl — no dependency on NPM or any other service. Reads the actual certificate
|
||||
# the server is presenting to the outside world.
|
||||
#
|
||||
# ── WHY DIRECT OPENSSL ────────────────────────────────────────────────────────────────────────
|
||||
# Catches real-world cert issues that API-based checks miss:
|
||||
# - Cert renewed in NPM but server not reloaded (old cert still serving)
|
||||
# - Wrong cert being served to external clients
|
||||
# - Cert chain issues not visible from the internal network
|
||||
# - NPM reporting healthy while the world sees an expired cert
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# SSL certificate expiry monitoring for all configured domains. Scheduled weekly
|
||||
# (Sunday 9am). Connects via openssl directly to each domain — not to NPM's API,
|
||||
# not to any internal check, but to the actual TLS handshake the outside world sees.
|
||||
#
|
||||
# ── BEHAVIOUR ─────────────────────────────────────────────────────────────────────────────────
|
||||
# Each domain is checked independently — they have independent certs.
|
||||
# Results per domain:
|
||||
# HEALTHY — > CERT_WARN_DAYS remaining — silent ✅
|
||||
# WARNING — <= CERT_WARN_DAYS remaining — notifies
|
||||
# CRITICAL — <= CERT_CRIT_DAYS remaining — notifies with urgency
|
||||
# FAILED — could not connect or parse cert — notifies
|
||||
# Per domain: HEALTHY (> CERT_WARN_DAYS remaining, silent) | WARNING (≤ CERT_WARN_DAYS)
|
||||
# | CRITICAL (≤ CERT_CRIT_DAYS) | FAILED (could not connect or parse cert).
|
||||
# Notifications batched by severity — one message lists all WARNING domains, a
|
||||
# separate message lists all CRITICAL domains. Not one notification per domain.
|
||||
#
|
||||
# Notifications batched per severity — one message per severity level, not per domain.
|
||||
# This is a monitor script — SILENT_MODE=false — output is the point.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_CERT_MONITOR_DOMAINS → CERT_MONITOR_DOMAINS.
|
||||
# Each server monitors its own domains — HOST1 monitors Gmer4Lfe.com etc.
|
||||
# Direct openssl, Not an API
|
||||
# API-based cert checks ask the certificate manager whether the cert is valid.
|
||||
# openssl checks ask the server what cert it is actually serving. These are not
|
||||
# the same question and the answers can differ. Catches: cert renewed in NPM but
|
||||
# server not reloaded (old cert still serving), wrong cert being served to external
|
||||
# clients, chain issues visible externally but not internally, NPM reporting healthy
|
||||
# while the outside world sees an expired cert.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent runs
|
||||
# detect_hosts() — correct domain list per host via MY_ID aliases
|
||||
# Empty array guard — warns and exits cleanly if no domains configured
|
||||
# CERT_TIMEOUT — openssl connects are time-limited per domain
|
||||
# validate_unraid_cmd — openssl and notify validated before use
|
||||
# Silent healthy certs — only problems produce visible output
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_CERT_MONITOR_DOMAINS — domains checked by this host
|
||||
# Aliased by detect_hosts() — script uses CERT_MONITOR_DOMAINS
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs producing duplicate notifications.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# CERT_WARN_DAYS — warn when cert expires within this many days (default 30)
|
||||
# CERT_CRIT_DAYS — critical alert within this many days (default 7)
|
||||
# CERT_TIMEOUT — seconds per domain before giving up (default 10)
|
||||
# Per-Host Domain List
|
||||
# detect_hosts() aliases HOST*_CERT_MONITOR_DOMAINS → CERT_MONITOR_DOMAINS.
|
||||
# Each server monitors its own domains only.
|
||||
#
|
||||
# Empty Array Guard
|
||||
# Warns and exits cleanly if CERT_MONITOR_DOMAINS is empty — no silent no-op.
|
||||
#
|
||||
# Connection Timeout
|
||||
# CERT_TIMEOUT caps each openssl connection attempt. One unreachable domain
|
||||
# does not block the remaining domains.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms openssl and notify script are present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_CERT_MONITOR_DOMAINS
|
||||
# Domains this host monitors. Each domain and subdomain is a separate entry —
|
||||
# they have independent certs. Aliased by detect_hosts() → CERT_MONITOR_DOMAINS.
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# CERT_WARN_DAYS
|
||||
# Days before expiry at which to send a warning notification. (default: 30)
|
||||
#
|
||||
# CERT_CRIT_DAYS
|
||||
# Days before expiry at which to send a critical notification. (default: 7)
|
||||
#
|
||||
# CERT_TIMEOUT
|
||||
# Seconds to wait per domain before declaring FAILED. (default: 10)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# cert_monitor.sh
|
||||
# Check all configured domains and notify on WARNING, CRITICAL, or FAILED.
|
||||
# Silent when all domains are healthy.
|
||||
#
|
||||
# cert_monitor.sh --dry-run
|
||||
# Check all domains and show results. No notifications sent regardless of result.
|
||||
#
|
||||
# cert_monitor.sh --status
|
||||
# Show domain list, warning thresholds, and timeout. Then exit.
|
||||
#
|
||||
# cert_monitor.sh --log
|
||||
# Verbose per-domain output during the run.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# cert_monitor.sh — normal run
|
||||
# cert_monitor.sh --dry-run — check certs and show results, no notifications
|
||||
# cert_monitor.sh --log — verbose output
|
||||
# cert_monitor.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -1,53 +1,86 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Emby Session Report ========================================
|
||||
# ============================= Emby Session Report ============================================
|
||||
# ==============================================================================================
|
||||
# Generates a usage report from the Emby media server via its API.
|
||||
# Queries activity logs and session history to produce a summary of what was
|
||||
# watched, by whom, and how over the configured report period.
|
||||
#
|
||||
# ── REPORT INCLUDES ───────────────────────────────────────────────────────────────────────────
|
||||
# Server info — name, version, uptime
|
||||
# Active sessions — current streams, direct play vs transcode
|
||||
# Library stats — movie, episode, song counts
|
||||
# Activity history — play events from the last EMBY_REPORT_DAYS days
|
||||
# Top content — most played items in the period (top EMBY_REPORT_TOP_N)
|
||||
# Most active users — who watched the most in the period
|
||||
# Transcode ratio — how often transcoding was needed vs direct play
|
||||
# Ramdisk status — current transcode location and usage
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Emby usage report via the Emby API. Scheduled weekly (Sunday 11am). Queries
|
||||
# activity logs and session history to produce a summary of what was watched,
|
||||
# by whom, and how often. No persistent state — queries fresh on every run.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases EMBY_URL and EMBY_API_KEY.
|
||||
# Each server reports on its own Emby instance automatically.
|
||||
# Reports: server info and uptime, active sessions and transcode ratio, library
|
||||
# counts (movies/episodes/songs), activity history for the last EMBY_REPORT_DAYS
|
||||
# days, top EMBY_REPORT_TOP_N content items, most active users, and ramdisk
|
||||
# transcode status.
|
||||
#
|
||||
# ── BEHAVIOUR ─────────────────────────────────────────────────────────────────────────────────
|
||||
# No persistent writes — queries API fresh each run.
|
||||
# This is a monitor/report script — SILENT_MODE=false — output is the point.
|
||||
# Silent when healthy (no notification on clean run).
|
||||
# Notifies only if transcoding is very high (>80% of streams) — may indicate config issue.
|
||||
# Notifies only if transcoding exceeds 80% of streams — may indicate a client
|
||||
# configuration issue. Silent on clean runs.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents duplicate reports running simultaneously
|
||||
# check_api() — verifies Emby reachable before queries
|
||||
# jq + curl validation — exits if either tool missing
|
||||
# validate_unraid_cmd — notify script validated before use
|
||||
# Per-section guards — API failure in one section does not abort others
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_EMBY_URL / HOST*_EMBY_API_KEY
|
||||
# Aliased by detect_hosts() — script uses EMBY_URL / EMBY_API_KEY
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate reports running simultaneously.
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# EMBY_REPORT_DAYS — days to include in the report period (default 7)
|
||||
# EMBY_REPORT_TOP_N — number of top content items to show (default 10)
|
||||
# RAMDISK_PATH — ramdisk mount path (for transcode status)
|
||||
# TRANSCODE_LINK — symlink path (for transcode location)
|
||||
# API Connectivity Check
|
||||
# check_api() verifies Emby is reachable before any queries. API failure in
|
||||
# one section does not abort the others — each section guards itself.
|
||||
#
|
||||
# Tool Validation
|
||||
# Checks for curl and jq at startup — exits with a clear error if either is missing.
|
||||
#
|
||||
# Per-Host Credentials
|
||||
# detect_hosts() aliases HOST*_EMBY_URL and HOST*_EMBY_API_KEY → EMBY_URL / EMBY_API_KEY.
|
||||
# Each server reports on its own Emby instance automatically.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_EMBY_URL
|
||||
# Emby server URL for this host. Aliased by detect_hosts() → EMBY_URL.
|
||||
#
|
||||
# HOST*_EMBY_API_KEY
|
||||
# Emby API key for this host. Aliased by detect_hosts() → EMBY_API_KEY.
|
||||
# Generate via Emby UI → Settings → API Keys → New API Key.
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# EMBY_REPORT_DAYS
|
||||
# Number of days to include in the activity history section. (default: 7)
|
||||
#
|
||||
# EMBY_REPORT_TOP_N
|
||||
# Number of top content items to show in the report. (default: 10)
|
||||
#
|
||||
# RAMDISK_PATH
|
||||
# Ramdisk mount path — used for transcode status reporting.
|
||||
#
|
||||
# TRANSCODE_LINK
|
||||
# Symlink path — used to determine current transcode location.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# emby_session_report.sh
|
||||
# Generate and send Emby usage report.
|
||||
#
|
||||
# emby_session_report.sh --dry-run
|
||||
# Test API connectivity and generate report output. No notification sent.
|
||||
#
|
||||
# emby_session_report.sh --status
|
||||
# Show Emby URL, API key (masked), and report configuration. Then exit.
|
||||
#
|
||||
# emby_session_report.sh --log
|
||||
# Verbose per-section output during report generation.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# emby_session_report.sh — generate report
|
||||
# emby_session_report.sh --dry-run — test API connectivity only, no notification
|
||||
# emby_session_report.sh --log — verbose output
|
||||
# emby_session_report.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
+65
-39
@@ -1,53 +1,79 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= SMART Health Monitor =======================================
|
||||
# ============================= SMART Health Monitor ===========================================
|
||||
# ==============================================================================================
|
||||
# Checks SMART health attributes for all drives on the system.
|
||||
# Reads data live from each drive via smartctl — no persistent writes.
|
||||
# Designed to run weekly as a scheduled report.
|
||||
#
|
||||
# ── MONITORED ATTRIBUTES ──────────────────────────────────────────────────────────────────────
|
||||
# Overall SMART status — PASSED/FAILED — immediate fail = drive is dying
|
||||
# Reallocated_Sector_Ct — bad sectors remapped — any > 0 is concerning
|
||||
# Current_Pending_Sector — sectors waiting for reallocation — any > 0 is concerning
|
||||
# Offline_Uncorrectable — sectors that could not be corrected — any > 0 is critical
|
||||
# Temperature_Celsius — vs thresholds from dynamix.cfg (or master.conf fallback)
|
||||
# Power_On_Hours — informational — drive age in days
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Drive SMART health monitoring for all drives on the system. Scheduled weekly
|
||||
# (Sunday 7am). Queries live SMART attributes via smartctl — no persistent writes.
|
||||
#
|
||||
# ── DRIVE DISCOVERY ───────────────────────────────────────────────────────────────────────────
|
||||
# Discovers drives automatically via /dev/sd* and /dev/nvme* — no config needed.
|
||||
# NVMe drives use different attribute names — detected and handled automatically.
|
||||
# HOST*_SMART_IGNORE_DRIVES skips specific drives (e.g. boot USB flash drive).
|
||||
# Monitored per drive: overall SMART status (PASSED/FAILED), Reallocated_Sector_Ct
|
||||
# (any > 0 is concerning), Current_Pending_Sector (any > 0 is concerning),
|
||||
# Offline_Uncorrectable (any > 0 is critical), Temperature_Celsius vs thresholds,
|
||||
# Power_On_Hours (informational). NVMe drives use different attribute names —
|
||||
# detected and handled automatically. Silent when all drives pass.
|
||||
#
|
||||
# ── TEMPERATURE THRESHOLDS ────────────────────────────────────────────────────────────────────
|
||||
# Reads hot/max/hotssd/maxssd from /boot/config/plugins/dynamix/dynamix.cfg at runtime.
|
||||
# Uses unRAID's own configured thresholds — no need to duplicate them here.
|
||||
# Falls back to SMART_TEMP_WARN / SMART_TEMP_CRIT from master.conf if dynamix.cfg not found.
|
||||
# Temperature thresholds read from /boot/config/plugins/dynamix/dynamix.cfg —
|
||||
# unRAID's own configured values. Falls back to SMART_TEMP_WARN / SMART_TEMP_CRIT
|
||||
# from master.conf if dynamix.cfg is not found.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_SMART_IGNORE_DRIVES → SMART_IGNORE_DRIVES.
|
||||
# Each server monitors its own drives with its own ignore list.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — smartctl calls are slow, prevent duplicate runs
|
||||
# detect_hosts() — correct ignore list per host via MY_ID aliases
|
||||
# validate_unraid_cmd — smartctl and notify validated before use
|
||||
# Silent healthy drives — only problems produce output
|
||||
# Silent healthy run — no notify when all drives pass
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent runs — smartctl calls are slow.
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_SMART_IGNORE_DRIVES — drives skipped in SMART monitoring
|
||||
# Aliased by detect_hosts() — script uses SMART_IGNORE_DRIVES
|
||||
# Per-Host Ignore List
|
||||
# detect_hosts() aliases HOST*_SMART_IGNORE_DRIVES → SMART_IGNORE_DRIVES.
|
||||
# Typically used to skip the boot USB flash drive (no meaningful SMART data).
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# SMART_TEMP_WARN — fallback warn threshold in °C (if dynamix.cfg not found)
|
||||
# SMART_TEMP_CRIT — fallback crit threshold in °C (if dynamix.cfg not found)
|
||||
# Automatic Drive Discovery
|
||||
# Scans /dev/sd* and /dev/nvme* on every run — no drive list to maintain.
|
||||
#
|
||||
# Dynamix Temperature Thresholds
|
||||
# Reads hot/max/hotssd/maxssd from dynamix.cfg so smart_health.sh and unRAID's
|
||||
# dashboard use the same thresholds. Falls back to master.conf values if not found.
|
||||
#
|
||||
# Notifications Validated
|
||||
# validate_unraid_cmd confirms smartctl and notify script are present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_SMART_IGNORE_DRIVES
|
||||
# Drives skipped in SMART monitoring. Aliased by detect_hosts() →
|
||||
# SMART_IGNORE_DRIVES. Typically includes the boot USB flash drive (sda).
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# SMART_TEMP_WARN
|
||||
# Fallback warn threshold in °C if dynamix.cfg not found. (default: 45)
|
||||
#
|
||||
# SMART_TEMP_CRIT
|
||||
# Fallback critical threshold in °C if dynamix.cfg not found. (default: 55)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# smart_health.sh
|
||||
# Query SMART attributes for all drives. Notify on any concerning results.
|
||||
# Silent when all drives pass.
|
||||
#
|
||||
# smart_health.sh --dry-run
|
||||
# Show which drives would be checked. No smartctl queries, no notifications.
|
||||
#
|
||||
# smart_health.sh --status
|
||||
# Show configured ignore list and temperature thresholds. Then exit.
|
||||
#
|
||||
# smart_health.sh --log
|
||||
# Verbose per-drive attribute output during the run.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# smart_health.sh — normal run
|
||||
# smart_health.sh --dry-run — show which drives would be checked
|
||||
# smart_health.sh --log — verbose output
|
||||
# smart_health.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,58 +2,101 @@
|
||||
# ==============================================================================================
|
||||
# ============================= System Tuning Monitor ==========================================
|
||||
# ==============================================================================================
|
||||
# Tracks inotify and php-fpm usage over time.
|
||||
# Snapshots written every 6 hours — read by sunday_morning_coffee_report.sh for weekly summary.
|
||||
# Schedule: 0 */6 * * * (every 6 hours via User Scripts)
|
||||
#
|
||||
# ── WHAT IT TRACKS ────────────────────────────────────────────────────────────────────────────
|
||||
# inotify instances:
|
||||
# Current in use vs kernel limit
|
||||
# % utilization — warns above INOTIFY_WARN_PCT (default 80%)
|
||||
# Top 5 consumers by instance count
|
||||
# Symptom of exhaustion: containers miss file events, downloads not detected,
|
||||
# Live TV stutter, library not updated
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# inotify and PHP-FPM utilisation tracking via time-series snapshots. Scheduled
|
||||
# every 6 hours (0 */6 * * *). Writes one bounded log entry per run to
|
||||
# TUNING_MONITOR_LOG. weekly_health_digest.sh reads this log to report peak,
|
||||
# average, and warning counts over the week.
|
||||
#
|
||||
# php-fpm workers:
|
||||
# Active workers vs PHP_MAX_CHILDREN limit
|
||||
# % utilization — warns above PHP_FPM_WARN_PCT (default 80%)
|
||||
# Symptom: unRAID WebGUI slowdowns or timeouts under load
|
||||
# Background snapshot script — no output when healthy. Warns (and notifies) only
|
||||
# when INOTIFY_WARN_PCT or PHP_FPM_WARN_PCT thresholds are exceeded.
|
||||
#
|
||||
# ── LOG FORMAT ────────────────────────────────────────────────────────────────────────────────
|
||||
# DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
|
||||
# Log trimmed to TUNING_LOG_RETENTION days on each write — bounded size.
|
||||
# inotify exhaustion symptoms: downloads complete but arrs don't detect them,
|
||||
# live TV stutter, library updates stop. PHP-FPM exhaustion symptoms: unRAID
|
||||
# WebGUI slowdowns or timeouts under load.
|
||||
#
|
||||
# ── WHAT THE WEEKLY REPORT SHOWS ──────────────────────────────────────────────────────────────
|
||||
# inotify: peak, average, warning count over the week
|
||||
# php-fpm: peak workers, average workers, warning count over the week
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SILENT BY DEFAULT ─────────────────────────────────────────────────────────────────────────
|
||||
# Background snapshot script — no output when healthy.
|
||||
# Warns to stderr when thresholds exceeded — visible in User Scripts output log.
|
||||
# Does NOT notify on every snapshot — only when threshold exceeded.
|
||||
# Each run snapshots:
|
||||
# inotify: instances in use vs INOTIFY_MAX_INSTANCES kernel limit.
|
||||
# Top 5 consumers by instance count. Warns above INOTIFY_WARN_PCT.
|
||||
# php-fpm: active workers vs PHP_MAX_CHILDREN limit.
|
||||
# Warns above PHP_FPM_WARN_PCT.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# Each server writes to its own DATA_DIR — no collision between servers.
|
||||
# MY_ID included in warning output for clarity in shared notification channels.
|
||||
# Log line format (one per run, trimmed to TUNING_LOG_RETENTION days):
|
||||
# DATE|TIME|INOTIFY_USED|INOTIFY_LIMIT|INOTIFY_PCT|INOTIFY_WARN|
|
||||
# PHPFPM_ACTIVE|PHPFPM_MAX|PHPFPM_PCT|PHPFPM_WARN
|
||||
# INOTIFY_WARN and PHPFPM_WARN are 1/0 flags. weekly_health_digest.sh counts
|
||||
# warnings over the week to show trend severity.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents overlapping 6-hour snapshots
|
||||
# root check — /proc/*/fd requires root access
|
||||
# atomic log write — tmp file + mv prevents partial writes on trim
|
||||
# validate_unraid — notify script validated before use
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# INOTIFY_WARN_PCT — warn threshold % (default 80)
|
||||
# PHP_FPM_WARN_PCT — warn threshold % (default 80)
|
||||
# PHP_MAX_CHILDREN — max php-fpm workers (set by php_fpm_max_children.sh)
|
||||
# TUNING_MONITOR_LOG — log file path
|
||||
# TUNING_LOG_RETENTION — days before old entries purged (default 30)
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents overlapping 6-hour snapshot runs.
|
||||
#
|
||||
# Root Enforcement
|
||||
# /proc/*/fd enumeration for inotify consumer counting requires root access.
|
||||
#
|
||||
# Atomic Log Write
|
||||
# Trim uses tmp file + mv — partial writes during log rotation cannot corrupt
|
||||
# the accumulated history.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# TUNING_MONITOR_LOG — DATA_DIR/tuning_monitor.db
|
||||
# One line per 6-hour snapshot. Trimmed to TUNING_LOG_RETENTION days on
|
||||
# each write — bounded size. Read by weekly_health_digest.sh for trend
|
||||
# reporting. Resets if DATA_DIR is cleared.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# INOTIFY_WARN_PCT
|
||||
# Warn if inotify instances exceed this percentage of the kernel limit. (default: 80)
|
||||
#
|
||||
# PHP_FPM_WARN_PCT
|
||||
# Warn if php-fpm active workers exceed this percentage of PHP_MAX_CHILDREN. (default: 80)
|
||||
#
|
||||
# PHP_MAX_CHILDREN
|
||||
# Maximum php-fpm workers — set by php_fpm_max_children.sh in unRAID_Essentials.
|
||||
#
|
||||
# TUNING_MONITOR_LOG
|
||||
# Log file path. (default: DATA_DIR/tuning_monitor.db)
|
||||
#
|
||||
# TUNING_LOG_RETENTION
|
||||
# Days before old entries are purged. (default: 30)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# system_tuning_monitor.sh
|
||||
# Take snapshot. Write to log. Warn (and notify) if thresholds exceeded.
|
||||
# Silent when healthy.
|
||||
#
|
||||
# system_tuning_monitor.sh --dry-run
|
||||
# Measure inotify and PHP-FPM utilisation and display results. No log write.
|
||||
#
|
||||
# system_tuning_monitor.sh --status
|
||||
# Show thresholds, log path, and retention. Then exit.
|
||||
#
|
||||
# system_tuning_monitor.sh --log
|
||||
# Verbose output during the snapshot including top inotify consumers.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# system_tuning_monitor.sh — normal snapshot run
|
||||
# system_tuning_monitor.sh --dry-run — measure and show, no log write
|
||||
# system_tuning_monitor.sh --log — verbose output
|
||||
# system_tuning_monitor.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -1,61 +1,115 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Health Digest ==============================================
|
||||
# ============================= Health Digest ==================================================
|
||||
# ==============================================================================================
|
||||
# Aggregates system health data from across the ecosystem into a single digest report.
|
||||
# Reads existing state files — no new writes.
|
||||
#
|
||||
# ── THREE PROFILES ────────────────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Full ecosystem health aggregation from existing state files. Scheduled daily
|
||||
# (8am). DIGEST_PROFILE controls when notifications actually send — the cron
|
||||
# schedule never changes, only the profile in master.conf.
|
||||
#
|
||||
# Reads state files from across the system (watchdog strikes, fallback state,
|
||||
# skip list, bandwidth history, transcode stats, cert status) and compiles them
|
||||
# into a single digest. Reads only — writes nothing, changes nothing.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three profiles — switch by changing DIGEST_PROFILE in master.conf:
|
||||
#
|
||||
# always — sends every run regardless of findings
|
||||
# schedule daily for a daily digest
|
||||
# Use: daily digest of everything, even when healthy
|
||||
#
|
||||
# smart — sends only if something worth reporting was found
|
||||
# runs every run but stays silent when all healthy
|
||||
# DIGEST_SMART_ON_* toggles control what triggers a send
|
||||
# smart — sends only when something worth reporting was found
|
||||
# Stays silent on clean days. DIGEST_SMART_ON_* toggles control
|
||||
# what triggers a send — all independently configurable.
|
||||
#
|
||||
# weekly — sends once per week on DIGEST_DAY regardless of schedule frequency
|
||||
# run daily, digest only fires on DIGEST_DAY (default Sunday)
|
||||
# weekly — sends once per week on DIGEST_DAY (default Sunday), silent all other days
|
||||
# Use: one weekly summary without daily noise
|
||||
#
|
||||
# The cron schedule stays the same regardless of profile — change DIGEST_PROFILE in
|
||||
# master.conf to switch behaviour. No cron changes needed.
|
||||
#
|
||||
# ── DATA SOURCES (reads only) ─────────────────────────────────────────────────────────────────
|
||||
# FALLBACK_STATE_FILE — current fallback state
|
||||
# SYS_WATCHDOG_FAILED_FILE — container skip list (manual intervention needed)
|
||||
# WATCHDOG_STATE_FILE — active container watchdog strikes
|
||||
# SYS_WATCHDOG_STATE_FILE — active system watchdog strikes
|
||||
# BANDWIDTH_LOG — yesterday's transfer totals
|
||||
# TRANSCODE_DAILY_LOG — weekly transcode statistics
|
||||
# CERT_MONITOR_DOMAINS — live SSL cert check via openssl
|
||||
# Data sources (reads only):
|
||||
# FALLBACK_STATE_FILE — current fallback state
|
||||
# SYS_WATCHDOG_FAILED_FILE — container skip list (manual intervention needed)
|
||||
# WATCHDOG_STATE_FILE — active container watchdog strikes
|
||||
# SYS_WATCHDOG_STATE_FILE — active system watchdog strikes
|
||||
# BANDWIDTH_LOG — yesterday's transfer totals
|
||||
# TRANSCODE_DAILY_LOG — weekly transcode statistics
|
||||
# CERT_MONITOR_DOMAINS — live SSL cert check via openssl
|
||||
# RAMDISK_PATH / TRANSCODE_LINK — current transcode location and usage
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases CERT_MONITOR_DOMAINS, RAMDISK_WARN_GB,
|
||||
# RAMDISK_SIZE, RAMDISK_LOW_GB and all other host-specific vars used in this report.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — report takes time, prevent duplicate runs
|
||||
# detect_hosts() — correct vars per host
|
||||
# validate_unraid_cmd — notify and openssl validated before use
|
||||
# Per-section guards — missing state file skipped cleanly
|
||||
# Silent smart profile — completely silent when nothing to report
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate reports — report generation takes time.
|
||||
#
|
||||
# Per-Host Variables
|
||||
# detect_hosts() aliases CERT_MONITOR_DOMAINS, RAMDISK_WARN_GB, RAMDISK_SIZE,
|
||||
# RAMDISK_LOW_GB, and all other host-specific vars used in the report.
|
||||
#
|
||||
# Per-Section Guards
|
||||
# Each data source section checks whether its state file exists before reading.
|
||||
# A missing state file is skipped cleanly — it does not abort the report.
|
||||
#
|
||||
# Silent Smart Profile
|
||||
# smart profile produces no output and no notification when nothing worth
|
||||
# reporting is found.
|
||||
#
|
||||
# Notifications Validated
|
||||
# validate_unraid_cmd confirms notify and openssl are present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DIGEST_PROFILE
|
||||
# Notification frequency: always | smart | weekly. (default: weekly)
|
||||
#
|
||||
# DIGEST_DAY
|
||||
# Day name for weekly profile — must match `date +%A` output. (default: Sunday)
|
||||
#
|
||||
# DIGEST_SMART_ON_WATCHDOG
|
||||
# Send smart profile notification if any active watchdog strikes. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_FALLBACK
|
||||
# Send smart profile notification if fallback state is not NORMAL. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_CERT_WARN
|
||||
# Send smart profile notification if any cert is within CERT_WARN_DAYS. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_BANDWIDTH
|
||||
# Send smart profile notification if any transfer exceeded BANDWIDTH_WARN_GB. (default: true)
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# DIGEST_PROFILE — always | smart | weekly
|
||||
# DIGEST_DAY — day name for weekly profile (e.g. Sunday)
|
||||
# DIGEST_SMART_ON_WATCHDOG — send on active watchdog strikes
|
||||
# DIGEST_SMART_ON_FALLBACK — send on non-NORMAL fallback state
|
||||
# DIGEST_SMART_ON_CERT_WARN — send on cert warning
|
||||
# DIGEST_SMART_ON_BANDWIDTH — send on high bandwidth day
|
||||
# CERT_WARN_DAYS / CERT_CRIT_DAYS / CERT_TIMEOUT
|
||||
# Cert check thresholds — shared with cert_monitor.sh.
|
||||
#
|
||||
# BANDWIDTH_WARN_GB
|
||||
# High-transfer threshold — shared with bandwidth_monitor.sh.
|
||||
#
|
||||
# TRANSCODE_DAILY_LOG
|
||||
# Path to the transcode statistics log.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# weekly_health_digest.sh
|
||||
# Run digest. DIGEST_PROFILE determines whether a notification is sent.
|
||||
#
|
||||
# weekly_health_digest.sh --dry-run
|
||||
# Generate and display digest output. No notification sent regardless of profile.
|
||||
#
|
||||
# weekly_health_digest.sh --status
|
||||
# Show profile, day, and smart trigger configuration. Then exit.
|
||||
#
|
||||
# weekly_health_digest.sh --log
|
||||
# Verbose per-section output during digest generation.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# weekly_health_digest.sh — normal run
|
||||
# weekly_health_digest.sh --dry-run — generate report, no notification
|
||||
# weekly_health_digest.sh --log — verbose output
|
||||
# weekly_health_digest.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -1,53 +1,112 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= ZFS Memory Snapshot ========================================
|
||||
# ============================= ZFS Memory Snapshot ============================================
|
||||
# ==============================================================================================
|
||||
# Weekly ZFS pool health and memory diagnostic report.
|
||||
# Combines ZFS pool status, ARC statistics, memory summary, Docker memory usage
|
||||
# and kernel pressure into a single report. Informational only — no action taken.
|
||||
# system_watchdog.sh handles threshold-based intervention.
|
||||
#
|
||||
# ── WHAT IT REPORTS ───────────────────────────────────────────────────────────────────────────
|
||||
# ZFS pool health — status, state, errors per pool (excluding ignored pools)
|
||||
# ARC statistics — current size, max, utilization %, metadata pressure
|
||||
# Memory status — total/free/available RAM vs thresholds
|
||||
# Docker memory — top ZFS_REPORT_DOCKER_TOP containers by memory usage
|
||||
# Kernel pressure — vmstat snapshot (3 samples)
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Weekly ZFS pool health and memory diagnostic report. Scheduled Sunday 6am —
|
||||
# first in the Sunday monitoring block, before other scripts run. Informational
|
||||
# only — system_watchdog.sh handles threshold-based intervention.
|
||||
#
|
||||
# ── OUTPUT ────────────────────────────────────────────────────────────────────────────────────
|
||||
# Output goes to both console and ZFS_REPORT_LOG for later review.
|
||||
# In dry-run mode — console only, nothing written to log.
|
||||
# Notifies if any warning thresholds are exceeded.
|
||||
# Silent when all healthy — only problems produce output.
|
||||
# Combines ZFS pool status, ARC statistics, Docker memory usage, and kernel
|
||||
# memory pressure into a single snapshot. Output goes to both console (for User
|
||||
# Scripts output log) and ZFS_REPORT_LOG for week-over-week comparison.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_ZFS_REPORT_IGNORE_POOLS → ZFS_REPORT_IGNORE_POOLS.
|
||||
# Each server ignores its own single-disk ZFS array pools — not the peer's.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — zpool + docker stats are slow, prevent duplicates
|
||||
# detect_hosts() — correct pool ignore list per host
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# DOCKER_TIMEOUT — docker stats protected against hung daemon
|
||||
# ZFS not available — skips pool and ARC sections gracefully
|
||||
# Docker not available — skips container section gracefully
|
||||
# Five report sections (each skips gracefully if its data source is unavailable):
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_ZFS_REPORT_IGNORE_POOLS — pools excluded from health reporting
|
||||
# Aliased by detect_hosts() — script uses ZFS_REPORT_IGNORE_POOLS
|
||||
# ZFS pool health — status, state, errors per pool. Pools in
|
||||
# ZFS_REPORT_IGNORE_POOLS excluded from the report
|
||||
# (still fully monitored by unRAID — report-only exclusion).
|
||||
# ARC statistics — current ARC vs max, metadata pressure, hit rate.
|
||||
# Warns if ARC utilisation exceeds ZFS_REPORT_ARC_WARN_PCT.
|
||||
# Memory status — total, free, available RAM.
|
||||
# Warns if free < ZFS_REPORT_FREE_WARN_GB or
|
||||
# available < ZFS_REPORT_AVAIL_WARN_GB.
|
||||
# Docker memory — top ZFS_REPORT_DOCKER_TOP containers by memory usage.
|
||||
# Useful for spotting containers approaching watchdog limits.
|
||||
# Kernel pressure — vmstat snapshot (3 samples).
|
||||
#
|
||||
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
|
||||
# ZFS_REPORT_LOG — log file path for weekly reports
|
||||
# ZFS_REPORT_ARC_WARN_PCT — warn if ARC using more than this % of max
|
||||
# ZFS_REPORT_FREE_WARN_GB — warn if less than this GB free RAM
|
||||
# ZFS_REPORT_AVAIL_WARN_GB — warn if less than this GB available RAM
|
||||
# ZFS_REPORT_DOCKER_TOP — how many top Docker containers to show
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate runs — zpool and docker stats are slow.
|
||||
#
|
||||
# Per-Host Pool Ignore List
|
||||
# detect_hosts() aliases HOST*_ZFS_REPORT_IGNORE_POOLS → ZFS_REPORT_IGNORE_POOLS.
|
||||
# Single-disk JBOD members excluded from report noise per server.
|
||||
#
|
||||
# ZFS Availability Guard
|
||||
# Skips pool and ARC sections gracefully if ZFS is not available on this server.
|
||||
#
|
||||
# Docker Availability Guard
|
||||
# Skips container memory section gracefully if Docker is not responding.
|
||||
#
|
||||
# Docker Stats Timeout
|
||||
# DOCKER_TIMEOUT caps docker stats calls. A hung daemon does not block the report.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ZFS_REPORT_LOG — /var/log/zfs-weekly-health.log (tmpfs, resets on reboot)
|
||||
# Weekly report written here for comparison across runs. Open the log to
|
||||
# see pool health trend week over week without remembering last week's values.
|
||||
# In dry-run mode, console only — nothing written.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_ZFS_REPORT_IGNORE_POOLS
|
||||
# Pools excluded from health reporting. Single-disk JBOD members generate
|
||||
# expected high-usage warnings — exclude them to reduce report noise.
|
||||
# Aliased by detect_hosts() → ZFS_REPORT_IGNORE_POOLS.
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# ZFS_REPORT_LOG
|
||||
# Log file path for weekly reports. (default: /var/log/zfs-weekly-health.log)
|
||||
#
|
||||
# ZFS_REPORT_ARC_WARN_PCT
|
||||
# Warn if ARC is using more than this percentage of its configured max. (default: 90)
|
||||
#
|
||||
# ZFS_REPORT_FREE_WARN_GB
|
||||
# Warn if free RAM is below this threshold in GB. (default: 10)
|
||||
#
|
||||
# ZFS_REPORT_AVAIL_WARN_GB
|
||||
# Warn if available RAM is below this threshold in GB. (default: 20)
|
||||
#
|
||||
# ZFS_REPORT_DOCKER_TOP
|
||||
# Number of top Docker containers by memory usage to include. (default: 10)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# zfs_memory_snapshot.sh
|
||||
# Generate report, write to ZFS_REPORT_LOG and console. Notify on warnings.
|
||||
#
|
||||
# zfs_memory_snapshot.sh --dry-run
|
||||
# Generate report to console only. No log write, no notifications.
|
||||
#
|
||||
# zfs_memory_snapshot.sh --status
|
||||
# Show pool ignore list and threshold configuration. Then exit.
|
||||
#
|
||||
# zfs_memory_snapshot.sh --log
|
||||
# Verbose output during report generation.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# zfs_memory_snapshot.sh — normal report (writes to log)
|
||||
# zfs_memory_snapshot.sh --dry-run — console only, no log write
|
||||
# zfs_memory_snapshot.sh --log — verbose output
|
||||
# zfs_memory_snapshot.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user