Varaverk/Monitors/README-Monitors.md

# Monitors

Watch and report only. Scripts in this folder never take action — they observe, measure, and notify. Intervention is handled by other parts of the ecosystem.

```
unRAID_Essentials/   — acts on the system (restarts, reboots, stops)
Docker_Essentials/   — acts on containers (watchdog, restarts)
Monitors/            — observes and reports (this folder)
```

All monitor scripts are safe to run at any time. None of them write to flash drives except `bandwidth_monitor.sh` which makes one bounded append per rsync run. All other monitors are read-only operations.

---

## Scripts

### `cert_monitor.sh`

Monitors SSL certificate expiry for all configured domains.

```bash
# Scheduled as: 0 9 * * 0  (Sunday 9am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh
```

**How it works:**

Connects directly to each domain via `openssl s_client` and reads the certificate the server is actually presenting. This is different from checking your certificate files directly — it catches real-world issues that file-based checks miss:

- Certificate renewed but web server not reloaded — old cert still being served
- Wrong certificate being served for a domain
- Certificate chain issues invisible to the cert file itself

Each domain and subdomain is a separate entry. They have independent certificates — `gmer4lfe.com` and `auth.gmer4lfe.com` may expire on different dates.

**Notification behavior:**
- Silent when all certs are healthy
- One warning notification for all domains approaching `CERT_WARN_DAYS`
- One critical notification for all domains within `CERT_CRIT_DAYS`
- Notifications batched by severity — not one per domain

**Configuration:**
```bash
CERT_MONITOR_DOMAINS=(
    "Gmer4Lfe.com"
    "Gmer4Lfe.us"
    # "auth.Gmer4Lfe.com"    # add subdomains as separate entries
)
CERT_WARN_DAYS=30   # warn when this many days remaining
CERT_CRIT_DAYS=7    # critical when this many days remaining
CERT_TIMEOUT=10     # seconds before giving up per domain
```

---

### `smart_health.sh`

Checks SMART health attributes for all drives in the system.

```bash
# Scheduled as: 0 7 * * 0  (Sunday 7am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh
```

**What it checks per drive:**

| Attribute | Threshold | Meaning |
|-----------|-----------|---------|
| Reallocated_Sector_Ct | > 0 = warning | Bad sectors remapped — drive showing wear |
| Current_Pending_Sector | > 0 = warning | Sectors awaiting reallocation |
| Offline_Uncorrectable | > 0 = critical | Sectors that could not be corrected |
| Temperature_Celsius | SMART_TEMP_WARN/CRIT | Drive running hot |
| Power_On_Hours | informational | Drive age estimate |
| Overall health status | PASSED/FAILED | Drive's own self-assessment |

Drive discovery is automatic — `/dev/sd*` and `/dev/nvme*` are scanned on every run. No drive list to maintain.

**Why ignore the boot USB:**

unRAID boots from a USB flash drive that typically appears as `sda`. Flash drives either don't support SMART or report meaningless values. Add it to `SMART_IGNORE_DRIVES` to keep it out of the report.

**Notification behavior:**
- Silent when all drives are healthy
- One warning notification listing all drives with concerning attributes
- One critical notification if any drive has uncorrectable sectors

**Configuration:**
```bash
SMART_TEMP_WARN=45          # degrees C
SMART_TEMP_CRIT=55          # degrees C
SMART_IGNORE_DRIVES=(
    "sda"                   # boot USB — not meaningful to check
)
```

---

### `zfs_memory_snapshot.sh`

Weekly ZFS pool health and memory diagnostic report.

```bash
# Scheduled as: 0 6 * * 0  (Sunday 6am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/zfs_memory_snapshot.sh
```

**What it reports:**

- Pool status — ONLINE/DEGRADED/FAULTED per pool
- Pool overview — size, allocated, free, capacity, health
- ARC statistics — max, current, metadata usage, utilization %
- Memory status — total, free, available RAM vs thresholds
- Top N Docker containers by memory usage
- Kernel pressure snapshot via vmstat

**Informational only.** This script reports what it finds. `system_watchdog.sh` handles threshold-based intervention — ARC reclaim, reboot decisions, memory pressure response. The snapshot gives you the weekly picture; the watchdog handles emergencies.

**Output is written to both console and `ZFS_REPORT_LOG`** — the log file lets you compare pool health week over week without having to remember what last week's numbers were.

**Pool ignore list:**

Pools expected to run at high capacity can be excluded from health reporting. They remain fully monitored by unRAID — this only affects what appears in the weekly report.

```bash
ZFS_REPORT_IGNORE_POOLS=(
    "disk10"    # high usage expected
    "disk9"
    "disk8"
    "disk6"
    "disk5"
)
```

**Configuration:**
```bash
ZFS_REPORT_LOG="/var/log/zfs-weekly-health.log"
ZFS_REPORT_ARC_WARN_PCT=90      # warn if ARC utilization above this %
ZFS_REPORT_FREE_WARN_GB=10      # warn if free RAM below this GB
ZFS_REPORT_AVAIL_WARN_GB=20     # warn if available RAM below this GB
ZFS_REPORT_DOCKER_TOP=10        # top N Docker memory users to show
```

---

### `backup_verify.sh`

Verifies the rsync mirror is healthy by comparing random file checksums between local and remote servers.

```bash
# Scheduled as: 0 10 * * 0  (Sunday 10am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh
```

**Why this matters:**

`rsync.sh` copies files successfully. But does the copy match the original? `backup_verify.sh` answers that question by independently computing MD5 checksums on both sides and comparing them. It catches:

- Silent data corruption during transfer
- Files that transferred but were corrupted at rest
- Partial transfers that rsync reported as success
- Storage hardware issues on either server

**How it works:**

1. Randomly samples `BACKUP_VERIFY_SAMPLE` files per share (files larger than `BACKUP_VERIFY_MIN_SIZE`)
2. Computes MD5 checksum locally
3. SSHes to remote and computes MD5 checksum there
4. Compares results

**Result per file:**

| Result | Meaning |
|--------|---------|
| MATCH | Checksums identical — file correctly mirrored |
| MISMATCH | File exists on both but checksums differ — sync may have failed |
| MISSING | File exists locally but not on remote — not yet synced or deleted |

**Configuration:**
```bash
# Leave empty to use DAILY_SYNC_SHARES automatically
BACKUP_VERIFY_SHARES=(
    # /mnt/user/Movies
    # /mnt/user/Tv_Shows
)
BACKUP_VERIFY_SAMPLE=10     # files sampled per share per run
BACKUP_VERIFY_MIN_SIZE=1M   # skip files smaller than this
```

If `BACKUP_VERIFY_SHARES` is empty the script automatically uses `DAILY_SYNC_SHARES` — no additional configuration needed for the standard setup.

Uses the existing SSH keys already configured for rsync — no additional setup required.

---

### `bandwidth_monitor.sh`

Logs rsync transfer history and generates weekly summary reports. Called automatically by `rsync.sh` — you do not need to schedule the logging mode manually.

```bash
# Log mode — called automatically by rsync.sh after each successful sync
# No manual scheduling needed

# Report mode — run manually or schedule weekly
# Scheduled as: 0 11 * * 0  (Sunday 11am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/bandwidth_monitor.sh --report
```

**Two modes:**

**`--log-transfer profile duration status`** — called by `rsync.sh` after each sync. Appends one line to the log file and trims entries older than `BANDWIDTH_LOG_RETENTION` days. You never call this manually.

**`--report`** — reads the log file and generates a weekly summary showing per-profile run counts, average durations, last 7 days activity, and failure counts.

**Log format:**
```
YYYY-MM-DD|HH:MM|profile|duration_seconds|status
2026-04-14|01:23|arrs_stack|287|success
2026-04-14|01:31|critical-data|143|success
2026-04-14|02:15|movies|1847|failed
```

**Why this format:**

The log never parses rsync output. Earlier designs tried to extract bytes transferred from rsync's human-readable output — that approach breaks silently when rsync updates and changes its output format. The current format captures what's reliably available: profile, duration, and success/failure. This is version-proof and survives any rsync update.

**Flash drive design:**

The log lives on `/boot/` so it survives reboots. Each rsync run makes exactly one append and one trim — the file never grows beyond `BANDWIDTH_LOG_RETENTION` lines. Minimal flash wear.

**Configuration:**
```bash
BANDWIDTH_LOG="/boot/config/bandwidth_history.db"
BANDWIDTH_LOG_RETENTION=90  # days — file stays bounded
BANDWIDTH_WARN_GB=50        # flag days exceeding this in reports
                            # (note: current log tracks duration not bytes)
```

---

### `weekly_health_digest.sh`

Aggregates system health data from across the entire ecosystem into a single digest report.

```bash
# Scheduled as: 0 8 * * *  (8am daily — profile controls when it notifies)
/mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh
```

**The key design:** schedule it daily regardless of profile. The `DIGEST_PROFILE` setting in `Master.conf` controls when a notification is actually sent — not the cron schedule.

**Three profiles:**

| Profile | Behavior | Use When |
|---------|----------|----------|
| `always` | Sends every run | You want a daily health summary |
| `smart` | Sends only if something worth reporting | Quiet operation, alerts on issues |
| `weekly` | Sends once per week on `DIGEST_DAY` | Weekly digest, silent other days |

Switch profiles by changing `DIGEST_PROFILE` in `Master.conf` — no cron changes needed.

**Data sources — reads only, no writes:**

| Source | What it reads |
|--------|---------------|
| `/tmp/transcode_state.db` | Ramdisk symlink and usage |
| `/tmp/container_watchdog_state.db` | Active container watchdog strikes |
| `/tmp/system_watchdog_state.db` | Active system watchdog strikes |
| `/boot/config/failover_state.db` | Current failover state |
| `/boot/config/system_watchdog_failed.db` | Container skip list |
| `/boot/config/bandwidth_history.db` | Recent transfer totals |
| Live `openssl` connection | SSL cert days remaining per domain |

**Smart profile triggers:**
```bash
# Set true to include this check in smart mode's "worth reporting" decision
DIGEST_SMART_ON_WATCHDOG=true    # any active watchdog strikes
DIGEST_SMART_ON_FAILOVER=true    # failover state is not NORMAL
DIGEST_SMART_ON_CERT_WARN=true   # any cert under CERT_WARN_DAYS
DIGEST_SMART_ON_BANDWIDTH=true   # any transfer exceeded BANDWIDTH_WARN_GB
```

**Configuration:**
```bash
DIGEST_PROFILE="weekly"     # always | smart | weekly
DIGEST_DAY="Sunday"         # for weekly profile — must match date +%A output
```

---

### `emby_session_report.sh`

Weekly Emby usage report via the Emby API.

```bash
# Scheduled as: 0 11 * * 0  (Sunday 11am weekly)
/mnt/user/appdata/unraid_scripts/Monitors/emby_session_report.sh
```

**What it reports:**
- Active streams at time of run
- Stream breakdown — total, Live TV, transcoding, direct play
- Library counts — movies, episodes, songs
- Current ramdisk transcode usage and symlink state

**No persistent writes** — queries the Emby API fresh on every run. No log files, no state. Run it any time for a current snapshot.

**Requires an Emby API key:**
1. Open Emby Settings → API Keys
2. Generate a new key
3. Paste it into `Master.conf` as `EMBY_API_KEY`

**Configuration:**
```bash
EMBY_URL="http://localhost:8096"
EMBY_API_KEY=""          # get from Emby Settings → API Keys
EMBY_REPORT_DAYS=7       # report period in days
EMBY_REPORT_TOP_N=10     # top N content items to show
```

---

## Flash Drive Write Policy

unRAID boots from a USB flash drive. Flash drives have limited write cycles. The Monitors folder is designed with this in mind:

| Script | Writes to flash | Notes |
|--------|----------------|-------|
| `cert_monitor.sh` | Never | Read-only openssl checks |
| `smart_health.sh` | Never | Read-only smartctl checks |
| `zfs_memory_snapshot.sh` | Never | Writes to `/var/log/` (RAM disk) |
| `backup_verify.sh` | Never | SSH + MD5 comparison only |
| `bandwidth_monitor.sh` | One append + one trim per rsync run | Bounded — never exceeds retention days |
| `weekly_health_digest.sh` | Never | Reads existing state files only |
| `emby_session_report.sh` | Never | API queries only |

The only flash write in the entire Monitors folder is `bandwidth_monitor.sh` — and it's designed to be minimal and bounded.

---

## Recommended Schedule

```bash
# Daily
0 8 * * *    weekly_health_digest.sh     # profile controls when it notifies

# Weekly — Sunday morning block
0 6 * * 0    zfs_memory_snapshot.sh
0 7 * * 0    smart_health.sh
0 9 * * 0    cert_monitor.sh
0 10 * * 0   backup_verify.sh
0 11 * * 0   emby_session_report.sh
0 11 * * 0   bandwidth_monitor.sh --report

# Automatic — no scheduling needed
# bandwidth_monitor.sh --log-transfer is called by rsync.sh after each sync
```

The Sunday morning block runs after the nightly maintenance window — by the time the monitors run, the weekly restarts, log clears, and media management jobs have completed. The health snapshot reflects a freshly maintained system.

---

## --dry-run Support

All monitor scripts support `--dry-run`. In dry-run mode:

- Checks run and results are shown
- No notifications are sent
- No files are written

Useful for testing configuration changes before scheduling:

```bash
/mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh --dry-run
/mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh --dry-run
/mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh --dry-run
```