user_script_plug-in.sh: v2.6 full audit and update
- Remove ARRS FAILED/STALLED RECOVERY as standalone orchestrator — now in INTERMEDIATE_MAINTENANCE_SCRIPTS (called every 4 hours by intermediate_sync) - Remove WEBGUI RESTART standalone block — webgui_watchdog.sh now runs via SYSTEM_WATCHDOG_SCRIPTS chain inside system_watchdog.sh (every minute) - Remove CLEAR LOGS standalone block — now in WEEKLY_MAINTENANCE_SCRIPTS (called as part of Sunday weekly_sync_maintenance window) - Remove SUNDAY MONITOR BLOCK individual cron entries — all monitors now run via COFFEE_REPORT_SCRIPTS inside sunday_morning_coffee_report.sh - Add MONTHLY MAINTENANCE orchestrator block (0 3 * * * — self-gated to monthly) - Update WATCHDOG ORCHESTRATOR to document SYSTEM_WATCHDOG_SCRIPTS chain - Update INTERMEDIATE SYNC to document arrs_failed in INTERMEDIATE_MAINTENANCE_SCRIPTS - Update WEEKLY SYNC to document playback_aware_*.sh in WEEKLY_MAINTENANCE_SCRIPTS - Fix INITIAL RUN paths: Initial_run/ → Partnership/ - Add individual entries: storage_watchdog, network_watchdog, playback_aware_lidarr/ radarr/sonarr, mesh_monitor, ramdisk_stop, emby_to_lidarr/radarr/sonarr, gitea_ssh_setup, partnership_offboard, partnership_transfer - Update continuous_scripts_status path to Old_Arch_Still_Works/ with deprecation note - Rewrite QUICK REFERENCE to match actual current schedule
This commit is contained in:
Gmer4Lfe
+231
-123
@@ -25,10 +25,8 @@
|
||||
# weekly_sync_maintenance.sh 2:30am Sunday — clean sync + image updates
|
||||
# sunday_morning_coffee_report.sh 7am Sunday — full weekly digest
|
||||
# weekly_health_digest.sh 8am daily — profile-controlled health notification
|
||||
# webgui_watchdog.sh every 10 min — WebGUI availability watchdog
|
||||
# system_tuning_monitor.sh every 6 hours — inotify + php-fpm trend tracking
|
||||
# clear_logs.sh 5am Sunday — size-threshold log cleanup
|
||||
# [Sunday monitor block] 6am-11am Sun — certs, SMART, backups, reports
|
||||
# monthly_maintenance.sh 3am daily — uptime-gated monthly tasks (self-gates to monthly)
|
||||
#
|
||||
# ── INDIVIDUAL SCRIPTS ────────────────────────────────────────────────────────────────────────
|
||||
# Every child script is also listed below, individually.
|
||||
@@ -91,6 +89,18 @@
|
||||
# v2.5 — fallback.sh --stop: added to ARRAY_STOP_SCRIPTS as step 2 (after user_scripts_stop,
|
||||
# before rsync_stop) — fallback runs as a bare subprocess, not via /tmp/user.scripts,
|
||||
# so it is not caught by user_scripts_stop.sh.
|
||||
# v2.6 — monthly_maintenance.sh: uptime-gated orchestrator for heavy monthly tasks (ZFS scrub,
|
||||
# SMART long test). Schedule 0 3 * * * — script self-gates to monthly cadence.
|
||||
# storage_watchdog.sh + network_watchdog.sh: pool growth + connectivity watchdogs,
|
||||
# called via SYSTEM_WATCHDOG_SCRIPTS by system_watchdog.sh (watchdog chain, no cron).
|
||||
# webgui_watchdog.sh moved from standalone cron to SYSTEM_WATCHDOG_SCRIPTS chain.
|
||||
# arrs_failed_stalled_recovery.sh moved to INTERMEDIATE_MAINTENANCE_SCRIPTS (every 4hr).
|
||||
# clear_logs.sh moved to WEEKLY_MAINTENANCE_SCRIPTS (Sunday weekly window).
|
||||
# Sunday monitor block replaced by COFFEE_REPORT_SCRIPTS (sunday_morning_coffee_report).
|
||||
# Playback-aware discovery (lidarr/radarr/sonarr) added to WEEKLY_MAINTENANCE_SCRIPTS.
|
||||
# Partnership/: gitea_ssh_setup.sh, partnership_offboard.sh, partnership_transfer.sh.
|
||||
# Tools/: emby_to_lidarr_sync.sh, emby_to_radarr_sync.sh, emby_to_sonarr_sync.sh,
|
||||
# ramdisk_stop.sh. Monitors/: mesh_monitor.sh. Path fix: Initial_run/ → Partnership/.
|
||||
# ==============================================================================================
|
||||
|
||||
|
||||
@@ -161,7 +171,11 @@
|
||||
# Execution order (non-negotiable):
|
||||
# 1. resource_watchdog.sh free RAM and reduce CPU pressure BEFORE healing attempts
|
||||
# 2. docker_watchdog.sh restart containers with pressure already reduced
|
||||
# 3. system_watchdog.sh reboot if prior layers could not resolve the issue
|
||||
# 3. system_watchdog.sh storage + WebGUI + network checks via SYSTEM_WATCHDOG_SCRIPTS:
|
||||
# storage_watchdog.sh pool growth + runaway log detection
|
||||
# webgui_watchdog.sh WebGUI availability (nginx → php-fpm → emhttp)
|
||||
# network_watchdog.sh internet, DDNS, Tailscale, NPM proxy
|
||||
# 4. stability_watchdog.sh reboot if all else failed — last line of defense
|
||||
#
|
||||
# Startup grace: no action until uptime >= WATCHDOG_STARTUP_GRACE (600s). Exits early
|
||||
# so all three sub-scripts don't each log startup-grace skips on every minute.
|
||||
@@ -239,37 +253,20 @@
|
||||
# Remote API keys read via SSH on the fly — never stored.
|
||||
# 2. Rsync window (optional) INTERMEDIATE_SYNC_SHARES — empty by default, skipped
|
||||
# Add shares here if a subset needs mid-day propagation.
|
||||
# 3. INTERMEDIATE_MAINTENANCE_SCRIPTS — lidarr_missing_art.sh and any future 4-hour jobs.
|
||||
# lidarr_missing_art.sh is idempotent — skips existing files.
|
||||
# 3. INTERMEDIATE_MAINTENANCE_SCRIPTS — 4-hour jobs (add any script that needs sub-daily cadence).
|
||||
# arrs_failed_stalled_recovery.sh — blocklist + re-search failed
|
||||
# and stalled arr imports. Every 4 hours. Configured in master.conf.
|
||||
# lidarr_missing_art.sh — idempotent artwork fetch. Skips existing files.
|
||||
#
|
||||
# Why separate from daily: arr libraries need to converge more than once a day. If a remote
|
||||
# node adds something at 2am, waiting 23 hours means both arrs search for content already owned.
|
||||
# 4-hour cycles close that gap. Artwork catches up within 4 hours of a new album landing.
|
||||
# 4-hour cycles close that gap. Artwork and failed imports catch up within 4 hours.
|
||||
#
|
||||
# Silent on success — runs 6x/day, only failures produce notification.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/intermediate_sync_maintenance.sh
|
||||
|
||||
|
||||
# ── ARRS FAILED/STALLED RECOVERY ─────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 */6 * * * (every 6 hours)
|
||||
# Background: YES
|
||||
#
|
||||
# Arrs never retry importFailed items automatically — this script handles it while you sleep.
|
||||
# Configured arrs: Sonarr (HOST1+HOST2), Radarr (HOST1+HOST2), Lidarr (HOST1 only)
|
||||
#
|
||||
# Finds items with these statuses, acts on EACH:
|
||||
# importFailed downloaded but couldn't import → blocklist + remove + new search
|
||||
# importPending stuck in import queue → blocklist + remove + new search
|
||||
# error serious failure state → blocklist + remove + new search
|
||||
# stalled no seeders or no progress → blocklist + remove + new search
|
||||
#
|
||||
# Never touches: items with status "downloading" or "imported"
|
||||
# Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (6hr default) are skipped — arr may still retry.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh
|
||||
|
||||
|
||||
# ── DAILY SYNC MAINTENANCE ────────────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 1 * * * (1am daily — runs on BOTH servers independently)
|
||||
# Background: YES
|
||||
@@ -333,6 +330,13 @@
|
||||
# 8. Start local containers dependency order, new image, verify each container up
|
||||
# 9. docker_weekly_restart.sh restart less-critical services: NextCloud, AdGuard, Immich
|
||||
# 10. docker_update_remaining.sh pull latest images for all containers not in daily/weekly lists
|
||||
# 11. WEEKLY_MAINTENANCE_SCRIPTS — clear_logs.sh + playback_aware_*.sh discovery scripts.
|
||||
# playback_aware_lidarr_discovery.sh: 0–5 music adds/week
|
||||
# based on your Emby play history + Last.fm similar artists.
|
||||
# playback_aware_radarr_discovery.sh: 0–5 movie adds/week
|
||||
# based on recently watched + TMDB recommendations.
|
||||
# playback_aware_sonarr_discovery.sh: 0–3 TV adds/week
|
||||
# based on recently watched + TMDB recommendations.
|
||||
#
|
||||
# Why weekly not nightly for Emby: HOST2 builds warm thumbnail cache throughout the week.
|
||||
# Nightly sync resets that cache — cold loads every morning. Weekly: cache warm 6 days,
|
||||
@@ -341,6 +345,29 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh
|
||||
|
||||
|
||||
# ── MONTHLY MAINTENANCE ───────────────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 3 * * * (daily 3am check — script self-gates to monthly cadence)
|
||||
# Background: YES
|
||||
#
|
||||
# Runs heavy tasks that require a long-stable, settled system. Fires only when BOTH gates pass:
|
||||
# 1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS (default: 30 days)
|
||||
# 2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS ago (default: 30 days)
|
||||
#
|
||||
# Why uptime-gated not cron: a scheduled reboot resets uptime. Monthly tasks (ZFS scrub,
|
||||
# SMART extended tests) need a system that has been healthy for a full month — not one that
|
||||
# just rebooted. The daily 3am cron is just the trigger. The gates inside enforce the cadence.
|
||||
# Silent exit 0 when either gate is not met — calling daily is always safe.
|
||||
#
|
||||
# Configure via master.conf MONTHLY_MAINTENANCE_SCRIPTS. Scripts planned but not yet built:
|
||||
# zfs_pool_scrub.sh ZFS pool integrity scrub (Tools/)
|
||||
# smart_long_test.sh SMART extended drive health test (Tools/)
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh --force
|
||||
|
||||
|
||||
# ── SUNDAY MORNING COFFEE REPORT ─────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 7 * * 0 (Sunday 7am — after 2:30am maintenance completes)
|
||||
# Background: YES
|
||||
@@ -385,25 +412,6 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh
|
||||
|
||||
|
||||
# ── WEBGUI RESTART ────────────────────────────────────────────────────────────────────────────
|
||||
# Schedule: */10 * * * * (every 10 minutes)
|
||||
# Background: YES
|
||||
#
|
||||
# Monitors WebGUI availability and recovers it automatically when unresponsive.
|
||||
# Silent when healthy — 144 runs per day produce no output when WebGUI responds normally.
|
||||
# Notification fires on any restart so you know what recovered and when.
|
||||
#
|
||||
# Escalation (each step only runs if previous step's recheck still fails):
|
||||
# 1. curl http://localhost unresponsive?
|
||||
# 2. Restart nginx wait WEBGUI_NGINX_WAIT=15s, recheck
|
||||
# 3. Restart php-fpm wait WEBGUI_PHP_WAIT=10s, recheck
|
||||
# handles PHP worker saturation (WebGUI feels slow)
|
||||
# 4. Restart emhttp (core daemon) wait WEBGUI_EMHTTP_WAIT=30s, recheck
|
||||
# 5. All failed → notify warning, manual intervention needed
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/webgui_watchdog.sh
|
||||
|
||||
|
||||
# ── SYSTEM TUNING MONITOR ────────────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 */6 * * * (every 6 hours)
|
||||
# Background: YES
|
||||
@@ -423,34 +431,19 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh
|
||||
|
||||
|
||||
# ── CLEAR LOGS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Schedule: 0 5 * * 0 (Sunday 5am — after 2:30am maintenance, before 6am monitors)
|
||||
# Background: YES
|
||||
#
|
||||
# Clears system and Docker container logs using size thresholds.
|
||||
# Only clears logs large enough to be worth clearing — preserves recent history on small logs.
|
||||
#
|
||||
# System logs (/var/log/syslog, messages, dmesg): only if > LOG_MIN_SIZE_MB (10MB)
|
||||
# Docker container logs: only if > LOG_DOCKER_MAX_MB (100MB) per container
|
||||
# Active containers (Emby, SABnzbd) grow fastest — 100MB threshold catches them.
|
||||
# Truncation not rotation: > file keeps file descriptor valid for running services.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh
|
||||
|
||||
|
||||
# ── SUNDAY MONITOR BLOCK ─────────────────────────────────────────────────────────────────────
|
||||
# Schedule: See individual entries below — 6am through 11am Sunday
|
||||
# Background: YES on all
|
||||
# These monitors are now driven by sunday_morning_coffee_report.sh via COFFEE_REPORT_SCRIPTS.
|
||||
# No separate User Script entries needed — configure which monitors run in master.conf.
|
||||
# All monitors are read-only — observe and report, never act. Still safe to run individually.
|
||||
#
|
||||
# Create a separate User Script entry for each monitor. Results feed into the 7am coffee report.
|
||||
# All monitors are read-only — observe and report, never act. Safe to run any time.
|
||||
#
|
||||
# 0 6 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/zfs_memory_snapshot.sh
|
||||
# 0 7 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh
|
||||
# 0 9 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh
|
||||
# 0 10 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh
|
||||
# 0 11 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/emby_session_report.sh
|
||||
# 0 11 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/bandwidth_monitor.sh --report
|
||||
# Current COFFEE_REPORT_SCRIPTS (master.conf):
|
||||
# Monitors/zfs_memory_snapshot.sh ZFS pool health + ARC + Docker memory snapshot
|
||||
# Monitors/smart_health.sh drive SMART attributes — reallocated, pending, temp
|
||||
# Monitors/cert_monitor.sh SSL certificate expiry for all configured domains
|
||||
# Monitors/backup_verify.sh rsync mirror integrity via independent MD5 checksums
|
||||
# Monitors/bandwidth_monitor.sh weekly rsync transfer totals and per-share breakdown
|
||||
# Monitors/emby_session_report.sh Emby usage — streams, users, library, transcode ratio
|
||||
# Monitors/weekly_health_digest.sh aggregated digest — watchdogs, fallback, skip list
|
||||
|
||||
|
||||
# ==============================================================================================
|
||||
@@ -709,9 +702,10 @@
|
||||
# UNRAID ESSENTIALS
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass)
|
||||
# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass.
|
||||
# All 18 checks independently toggleable per host in host*.conf.
|
||||
# system_watchdog.sh — third layer in the watchdog chain: runs SYSTEM_WATCHDOG_SCRIPTS each cycle
|
||||
# Called by watchdog_orchestrator.sh — NOT scheduled directly. Single-pass.
|
||||
# Iterates SYSTEM_WATCHDOG_SCRIPTS sequentially: storage_watchdog → webgui_watchdog → network_watchdog.
|
||||
# Each sub-script is independent — a failure in one does not skip the others.
|
||||
#
|
||||
# Tier 1 CRITICAL — bypass ALL strikes, reboot immediately:
|
||||
# Docker daemon hung → attempt rc.docker restart → still hung → reboot
|
||||
@@ -736,6 +730,39 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/stability_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/stability_watchdog.sh --dry-run
|
||||
|
||||
# storage_watchdog.sh — pool growth + runaway log detection (single-pass, called by system_watchdog)
|
||||
# Part of SYSTEM_WATCHDOG_SCRIPTS — not scheduled directly. Called by system_watchdog.sh each cycle.
|
||||
#
|
||||
# Two complementary checks every cycle:
|
||||
# Growth rate (zero-config): du per container vs previous baseline — any container growing
|
||||
# more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan inside that dir.
|
||||
# New containers covered automatically from their second cycle. No per-container config.
|
||||
# Absolute log size: finds *.log files over WATCHDOG_APPDATA_LOG_MAX_GB anywhere in
|
||||
# WATCHDOG_APPDATA_PATHS. Catches logs already large but not actively growing.
|
||||
#
|
||||
# Strike system (same pattern as docker_watchdog CPU/HTTP strikes):
|
||||
# Strike 1 → warn + notify. Strike 2 → escalated notify. Strike 3 → action:
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS=true → truncate *.log in-place, clear strikes
|
||||
# WATCHDOG_APPDATA_TRUNCATE_LOGS=false → critical notify, hold strikes until resolved
|
||||
# Condition resolves (growth stops / log shrinks) → strikes auto-clear.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/storage_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/storage_watchdog.sh --dry-run
|
||||
|
||||
# network_watchdog.sh — internet + DDNS + Tailscale + NPM proxy checks (called by system_watchdog)
|
||||
# Part of SYSTEM_WATCHDOG_SCRIPTS — not scheduled directly. Called by system_watchdog.sh each cycle.
|
||||
#
|
||||
# Check 1 — Internet: curl external endpoint. Fail → alert + skip all remaining (prevents cascades).
|
||||
# Check 2 — DDNS (Cloudflare): public IP via ifconfig.me vs DNS record via dig @1.1.1.1.
|
||||
# Mismatch → restart DDNS container (triggers immediate Cloudflare record update) + notify.
|
||||
# Check 3 — Tailscale: tailscale status → Running → pass. Not running → notify (no restart — human).
|
||||
# Check 4 — NPM proxy (external): curl external URL through the full stack.
|
||||
# 2-strike system: Strike 1 = warn + notify. Strike 2 = restart NginxProxyManager + notify.
|
||||
# Strikes auto-clear when the external URL becomes reachable again.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/network_watchdog.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/network_watchdog.sh --dry-run
|
||||
|
||||
# resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator)
|
||||
# Reduces system load intelligently BEFORE docker_watchdog attempts container restarts.
|
||||
# Containers restarted into a resource-pressured system just fail again — this runs first.
|
||||
@@ -787,11 +814,15 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh
|
||||
|
||||
# webgui_watchdog.sh — WebGUI availability watchdog (see ORCHESTRATORS section above)
|
||||
# webgui_watchdog.sh — WebGUI availability watchdog (called by system_watchdog via SYSTEM_WATCHDOG_SCRIPTS)
|
||||
# Not scheduled directly — runs as part of the every-minute watchdog chain.
|
||||
# Escalation: nginx → php-fpm → emhttp, each with recheck before proceeding to next level.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/webgui_watchdog.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/webgui_watchdog.sh --status
|
||||
|
||||
# clear_logs.sh — size-threshold log cleanup (see ORCHESTRATORS section above)
|
||||
# clear_logs.sh — size-threshold log cleanup (called by weekly_sync_maintenance via WEEKLY_MAINTENANCE_SCRIPTS)
|
||||
# Not scheduled directly — runs as part of the Sunday 2:30am weekly window.
|
||||
# System logs only cleared if > LOG_MIN_SIZE_MB (10MB). Docker logs only if > LOG_DOCKER_MAX_MB (100MB).
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --status
|
||||
|
||||
@@ -939,10 +970,40 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/arr_sync.sh --blocklist-remove radarr <id>
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/arr_sync.sh --blocklist-list
|
||||
|
||||
# arrs_failed_stalled_recovery.sh — blocklist + re-search failed/stalled arr imports (see ORCHESTRATORS)
|
||||
# arrs_failed_stalled_recovery.sh — blocklist + re-search failed/stalled arr imports
|
||||
# Called every 4 hours by intermediate_sync_maintenance.sh via INTERMEDIATE_MAINTENANCE_SCRIPTS.
|
||||
# Finds importFailed / importPending / error / stalled items across all configured arrs.
|
||||
# Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (6hr) are skipped — arr may still self-retry.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh --status
|
||||
|
||||
# playback_aware_lidarr_discovery.sh — behavior-driven music discovery (runs via WEEKLY_MAINTENANCE_SCRIPTS)
|
||||
# Called by weekly_sync_maintenance.sh on Sunday. Can be run standalone.
|
||||
# Stage 1: score artists you actually played this week in Emby (play count + recency + user diversity).
|
||||
# Stage 2: run Last.fm artist.getSimilar on top seeds, score recommendations, add top results to Lidarr.
|
||||
# Goal: 0–5 meaningful adds per run. Not bulk import — only artists likely to match your taste.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_lidarr_discovery.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_lidarr_discovery.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_lidarr_discovery.sh
|
||||
|
||||
# playback_aware_radarr_discovery.sh — behavior-driven movie discovery (runs via WEEKLY_MAINTENANCE_SCRIPTS)
|
||||
# Called by weekly_sync_maintenance.sh on Sunday. Can be run standalone.
|
||||
# Stage 1: score recently watched movies in Emby, weighted by recency and TMDB rating.
|
||||
# Stage 2: run TMDB recommendations on top seeds, score candidates, add top results to Radarr.
|
||||
# Goal: 0–5 meaningful adds per run.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_radarr_discovery.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_radarr_discovery.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_radarr_discovery.sh
|
||||
|
||||
# playback_aware_sonarr_discovery.sh — behavior-driven TV discovery (runs via WEEKLY_MAINTENANCE_SCRIPTS)
|
||||
# Called by weekly_sync_maintenance.sh on Sunday. Can be run standalone.
|
||||
# Stage 1: score recently watched series in Emby (user diversity + recency + per-user-capped episodes).
|
||||
# Stage 2: run TMDB TV recommendations on top seeds, score candidates, add top results to Sonarr.
|
||||
# Goal: 0–3 meaningful adds per run.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_sonarr_discovery.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_sonarr_discovery.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_sonarr_discovery.sh
|
||||
|
||||
# lidarr_missing_art.sh — fetch missing album and artist artwork for the Lidarr music library
|
||||
# Called by intermediate_sync_maintenance.sh via INTERMEDIATE_MAINTENANCE_SCRIPTS.
|
||||
# HOST1 only — exits cleanly on HOST2 (HOST2 does not run Lidarr).
|
||||
@@ -1041,6 +1102,16 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh
|
||||
|
||||
# ramdisk_stop.sh — safely stop the transcode ramdisk without interrupting Emby
|
||||
# Flips the transcode symlink to the SSD fallback BEFORE unmounting so Emby continues writing.
|
||||
# Then unmounts the tmpfs and updates the state file.
|
||||
# Use before: planned reboots, ramdisk size changes, ramdisk troubleshooting.
|
||||
# ramdisk_setup.sh recreates and re-mounts on next array start.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/ramdisk_stop.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/ramdisk_stop.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/ramdisk_stop.sh
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# MONITORS
|
||||
@@ -1127,17 +1198,19 @@
|
||||
# system_tuning_monitor.sh — inotify + PHP-FPM trend tracking (see ORCHESTRATORS)
|
||||
# bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh --status
|
||||
|
||||
# continuous_scripts_status.sh — live dashboard for all three background processes
|
||||
# The single command that shows you what the entire continuous-running ecosystem is doing.
|
||||
# Shows for system_watchdog, docker_watchdog, and failover:
|
||||
# Running state, PID, uptime, approximate cycle count
|
||||
# Active strikes per check. Required container states. Memory-monitored container usage.
|
||||
# Skip list contents (any entry needs human attention).
|
||||
# Fallback state, active tiers, outage duration, remote Tailscale visibility.
|
||||
# No schedule — on demand only. Run any time you want a complete status snapshot.
|
||||
# mesh_monitor.sh — at-a-glance view of all servers in the mesh (read-only, on demand)
|
||||
# No schedule — run any time you want a current mesh snapshot. No SSH or API calls.
|
||||
# Shows each node: hostname, Tailscale IP, role (owner/mirror), services covered in fallback tiers.
|
||||
# Reads directly from all three conf files — does not use detect_hosts().
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/continuous_scripts_status.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/continuous_scripts_status.sh --log
|
||||
# bash /mnt/user/appdata/unraid_scripts/Monitors/mesh_monitor.sh
|
||||
|
||||
# continuous_scripts_status.sh — legacy live dashboard (Old_Arch_Still_Works/)
|
||||
# NOTE: watchdog architecture changed to single-pass in v2.4. This dashboard was written
|
||||
# for the continuous-loop design and is no longer kept current. Use --status flags on
|
||||
# individual scripts instead: docker_watchdog --status, stability_watchdog --status, etc.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Old_Arch_Still_Works/continuous_scripts_status.sh
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
@@ -1170,19 +1243,47 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --onboard --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --onboard
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --offboard --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --offboard
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --transfer --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh \
|
||||
# --transfer --confirm=i-understand-this-transfers-ownership
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --unblock hostname
|
||||
|
||||
# partnership_offboard.sh — cleanly end a partnership (standalone version of --offboard)
|
||||
# Role is detected automatically — run on either server. Owner path runs the full sequence:
|
||||
# stop rsync → final sync → reconfigure mirror WebUIs → disable sync → local cleanup →
|
||||
# Tailscale removal. Mirror path handles local side and signals owner.
|
||||
# Deferred finalisation: if remote unreachable, completes automatically when reachable.
|
||||
# ALWAYS --dry-run first. 10-second countdown before live execution.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_offboard.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_offboard.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_offboard.sh
|
||||
|
||||
# partnership_transfer.sh — transfer ownership from current owner to current mirror
|
||||
# Owner only — mirror cannot initiate. Config and WebUI targets updated; containers stay.
|
||||
# Health strike system verifies both servers healthy before proceeding.
|
||||
# Requires confirmation string: "i-understand-this-transfers-ownership" (long by design).
|
||||
# After transfer: sync direction reverses on the next fallback/critical_sync cycle.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_transfer.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_transfer.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_transfer.sh \
|
||||
# --confirm=i-understand-this-transfers-ownership
|
||||
|
||||
# gitea_ssh_setup.sh — generate Gitea SSH keypair and register it with Gitea via API
|
||||
# Idempotent: skips generation if key exists, skips registration if same key already registered.
|
||||
# Use --force to regenerate and re-register (e.g. after Gitea reinstall or key compromise).
|
||||
# Called during onboard setup or when re-keying a server. Run after Gitea container is up.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh --force
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
# INITIAL RUN
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Run ONCE when setting up a new server or joining a new partnership. Not scheduled.
|
||||
# All initial-run scripts now live in Partnership/ alongside the lifecycle scripts they set up.
|
||||
|
||||
# ssh_setup.sh — generate the rsync SSH keypair and install it on the remote server
|
||||
# Key named from hostname: unRAID-Gmer4Lfe → gmer4lfe_rsync_automation
|
||||
@@ -1194,11 +1295,11 @@
|
||||
# At SSH_MAX_STRIKES (5): notify + return exit 2 for caller to escalate.
|
||||
# Strikes reset after SSH_STRIKE_RESET_HRS (24) of clean connectivity.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --force
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --validate
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --force
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --validate
|
||||
|
||||
# partnership_onboard.sh — full partnership setup orchestrator (run once to join a partner server)
|
||||
# Orchestrates the complete setup sequence in order:
|
||||
@@ -1213,10 +1314,10 @@
|
||||
# PARTNERSHIP_OWNER_HOST correct in master.conf.
|
||||
# ALWAYS --dry-run first.
|
||||
#
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh --skip-ssh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh --skip-arr-sync
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --skip-ssh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --skip-arr-sync
|
||||
|
||||
|
||||
# ──────────────────────────────────────────────────────────────────────────────────────────────
|
||||
@@ -1273,6 +1374,26 @@
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh poolname
|
||||
|
||||
# emby_to_lidarr_sync.sh — one-shot bootstrap: add all played Emby artists to Lidarr
|
||||
# Finds artists played on Emby that are not tracked in Lidarr and adds them. No scoring.
|
||||
# Run once after Lidarr is set up, or any time you suspect gaps between what you listen to
|
||||
# and what Lidarr monitors. Use --days N to limit lookback window.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_lidarr_sync.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_lidarr_sync.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_lidarr_sync.sh
|
||||
|
||||
# emby_to_radarr_sync.sh — one-shot bootstrap: add all played Emby movies to Radarr
|
||||
# Same pattern as emby_to_lidarr_sync. Run once to ensure everything you've watched is monitored.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_radarr_sync.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_radarr_sync.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_radarr_sync.sh
|
||||
|
||||
# emby_to_sonarr_sync.sh — one-shot bootstrap: add all played Emby series to Sonarr
|
||||
# Same pattern as emby_to_lidarr_sync. Run once to ensure all watched shows are monitored.
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_sonarr_sync.sh --dry-run
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_sonarr_sync.sh --status
|
||||
# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_sonarr_sync.sh
|
||||
|
||||
# recreate_shares.sh — create share directories on disks after fresh install or rebuild
|
||||
# Use when: share .cfg files restored to fresh install but directories missing on disks.
|
||||
# Reads /boot/config/shares/*.cfg, parses shareInclude list, creates /mnt/diskN/sharename.
|
||||
@@ -1296,53 +1417,40 @@
|
||||
#
|
||||
# * * * * * every minute:
|
||||
# Orchestrators/watchdog_orchestrator.sh
|
||||
# └─ resource_watchdog → docker_watchdog → system_watchdog (storage/webgui/network) → stability_watchdog
|
||||
#
|
||||
# */7 * * * * every 7 minutes:
|
||||
# Orchestrators/transcode_management.sh
|
||||
#
|
||||
# */10 * * * * every 10 minutes:
|
||||
# Watchdogs/System/webgui_watchdog.sh
|
||||
#
|
||||
# */30 * * * * every 30 minutes:
|
||||
# Orchestrators/critical_sync_maintenance.sh
|
||||
#
|
||||
# */30 * * * * every 30 minutes:
|
||||
# Rsync/rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback
|
||||
#
|
||||
# 0 */4 * * * every 4 hours:
|
||||
# Orchestrators/intermediate_sync_maintenance.sh
|
||||
# └─ arr_sync → arrs_failed_stalled_recovery + lidarr_missing_art (INTERMEDIATE_MAINTENANCE_SCRIPTS)
|
||||
#
|
||||
# 0 */6 * * * every 6 hours:
|
||||
# Media/arrs_failed_stalled_recovery.sh
|
||||
# Monitors/system_tuning_monitor.sh
|
||||
#
|
||||
# 0 1 * * * 1am daily:
|
||||
# Orchestrators/daily_sync_maintenance.sh
|
||||
#
|
||||
# 0 5 * * 0 Sunday 5am:
|
||||
# unRAID_Essentials/clear_logs.sh
|
||||
#
|
||||
# 0 6 * * 0 Sunday 6am:
|
||||
# Monitors/zfs_memory_snapshot.sh
|
||||
# 0 3 * * * 3am daily (fires monthly — self-gated by uptime + interval):
|
||||
# Orchestrators/monthly_maintenance.sh
|
||||
#
|
||||
# 0 7 * * 0 Sunday 7am:
|
||||
# Orchestrators/sunday_morning_coffee_report.sh
|
||||
# Monitors/smart_health.sh
|
||||
# └─ runs all monitors via COFFEE_REPORT_SCRIPTS (master.conf):
|
||||
# zfs_memory_snapshot / smart_health / cert_monitor / backup_verify
|
||||
# bandwidth_monitor / emby_session_report / weekly_health_digest
|
||||
#
|
||||
# 0 8 * * * 8am daily:
|
||||
# Monitors/weekly_health_digest.sh
|
||||
#
|
||||
# 0 9 * * 0 Sunday 9am:
|
||||
# Monitors/cert_monitor.sh
|
||||
#
|
||||
# 0 10 * * 0 Sunday 10am:
|
||||
# Monitors/backup_verify.sh
|
||||
#
|
||||
# 0 11 * * 0 Sunday 11am:
|
||||
# Monitors/emby_session_report.sh
|
||||
# Monitors/bandwidth_monitor.sh --report
|
||||
#
|
||||
# 30 2 * * 0 Sunday 2:30am:
|
||||
# Orchestrators/weekly_sync_maintenance.sh
|
||||
# └─ runs WEEKLY_MAINTENANCE_SCRIPTS after sync:
|
||||
# clear_logs / playback_aware_lidarr / playback_aware_radarr / playback_aware_sonarr
|
||||
#
|
||||
# ==============================================================================================
|
||||
Reference in New Issue
Block a user