From 0e13d8772d10f322849cadab0ed99a3063bd3798 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 23 May 2026 20:57:47 -0400 Subject: [PATCH] user_script_plug-in.sh: v2.6 full audit and update MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove ARRS FAILED/STALLED RECOVERY as standalone orchestrator — now in INTERMEDIATE_MAINTENANCE_SCRIPTS (called every 4 hours by intermediate_sync) - Remove WEBGUI RESTART standalone block — webgui_watchdog.sh now runs via SYSTEM_WATCHDOG_SCRIPTS chain inside system_watchdog.sh (every minute) - Remove CLEAR LOGS standalone block — now in WEEKLY_MAINTENANCE_SCRIPTS (called as part of Sunday weekly_sync_maintenance window) - Remove SUNDAY MONITOR BLOCK individual cron entries — all monitors now run via COFFEE_REPORT_SCRIPTS inside sunday_morning_coffee_report.sh - Add MONTHLY MAINTENANCE orchestrator block (0 3 * * * — self-gated to monthly) - Update WATCHDOG ORCHESTRATOR to document SYSTEM_WATCHDOG_SCRIPTS chain - Update INTERMEDIATE SYNC to document arrs_failed in INTERMEDIATE_MAINTENANCE_SCRIPTS - Update WEEKLY SYNC to document playback_aware_*.sh in WEEKLY_MAINTENANCE_SCRIPTS - Fix INITIAL RUN paths: Initial_run/ → Partnership/ - Add individual entries: storage_watchdog, network_watchdog, playback_aware_lidarr/ radarr/sonarr, mesh_monitor, ramdisk_stop, emby_to_lidarr/radarr/sonarr, gitea_ssh_setup, partnership_offboard, partnership_transfer - Update continuous_scripts_status path to Old_Arch_Still_Works/ with deprecation note - Rewrite QUICK REFERENCE to match actual current schedule --- user_script_plug-in.sh | 354 +++++++++++++++++++++++++++-------------- 1 file changed, 231 insertions(+), 123 deletions(-) diff --git a/user_script_plug-in.sh b/user_script_plug-in.sh index 5c2f24e..5ce8f4b 100644 --- a/user_script_plug-in.sh +++ b/user_script_plug-in.sh @@ -25,10 +25,8 @@ # weekly_sync_maintenance.sh 2:30am Sunday — clean sync + image updates # sunday_morning_coffee_report.sh 7am Sunday — full weekly digest # weekly_health_digest.sh 8am daily — profile-controlled health notification -# webgui_watchdog.sh every 10 min — WebGUI availability watchdog # system_tuning_monitor.sh every 6 hours — inotify + php-fpm trend tracking -# clear_logs.sh 5am Sunday — size-threshold log cleanup -# [Sunday monitor block] 6am-11am Sun — certs, SMART, backups, reports +# monthly_maintenance.sh 3am daily — uptime-gated monthly tasks (self-gates to monthly) # # ── INDIVIDUAL SCRIPTS ──────────────────────────────────────────────────────────────────────── # Every child script is also listed below, individually. @@ -91,6 +89,18 @@ # v2.5 — fallback.sh --stop: added to ARRAY_STOP_SCRIPTS as step 2 (after user_scripts_stop, # before rsync_stop) — fallback runs as a bare subprocess, not via /tmp/user.scripts, # so it is not caught by user_scripts_stop.sh. +# v2.6 — monthly_maintenance.sh: uptime-gated orchestrator for heavy monthly tasks (ZFS scrub, +# SMART long test). Schedule 0 3 * * * — script self-gates to monthly cadence. +# storage_watchdog.sh + network_watchdog.sh: pool growth + connectivity watchdogs, +# called via SYSTEM_WATCHDOG_SCRIPTS by system_watchdog.sh (watchdog chain, no cron). +# webgui_watchdog.sh moved from standalone cron to SYSTEM_WATCHDOG_SCRIPTS chain. +# arrs_failed_stalled_recovery.sh moved to INTERMEDIATE_MAINTENANCE_SCRIPTS (every 4hr). +# clear_logs.sh moved to WEEKLY_MAINTENANCE_SCRIPTS (Sunday weekly window). +# Sunday monitor block replaced by COFFEE_REPORT_SCRIPTS (sunday_morning_coffee_report). +# Playback-aware discovery (lidarr/radarr/sonarr) added to WEEKLY_MAINTENANCE_SCRIPTS. +# Partnership/: gitea_ssh_setup.sh, partnership_offboard.sh, partnership_transfer.sh. +# Tools/: emby_to_lidarr_sync.sh, emby_to_radarr_sync.sh, emby_to_sonarr_sync.sh, +# ramdisk_stop.sh. Monitors/: mesh_monitor.sh. Path fix: Initial_run/ → Partnership/. # ============================================================================================== @@ -161,7 +171,11 @@ # Execution order (non-negotiable): # 1. resource_watchdog.sh free RAM and reduce CPU pressure BEFORE healing attempts # 2. docker_watchdog.sh restart containers with pressure already reduced -# 3. system_watchdog.sh reboot if prior layers could not resolve the issue +# 3. system_watchdog.sh storage + WebGUI + network checks via SYSTEM_WATCHDOG_SCRIPTS: +# storage_watchdog.sh pool growth + runaway log detection +# webgui_watchdog.sh WebGUI availability (nginx → php-fpm → emhttp) +# network_watchdog.sh internet, DDNS, Tailscale, NPM proxy +# 4. stability_watchdog.sh reboot if all else failed — last line of defense # # Startup grace: no action until uptime >= WATCHDOG_STARTUP_GRACE (600s). Exits early # so all three sub-scripts don't each log startup-grace skips on every minute. @@ -239,37 +253,20 @@ # Remote API keys read via SSH on the fly — never stored. # 2. Rsync window (optional) INTERMEDIATE_SYNC_SHARES — empty by default, skipped # Add shares here if a subset needs mid-day propagation. -# 3. INTERMEDIATE_MAINTENANCE_SCRIPTS — lidarr_missing_art.sh and any future 4-hour jobs. -# lidarr_missing_art.sh is idempotent — skips existing files. +# 3. INTERMEDIATE_MAINTENANCE_SCRIPTS — 4-hour jobs (add any script that needs sub-daily cadence). +# arrs_failed_stalled_recovery.sh — blocklist + re-search failed +# and stalled arr imports. Every 4 hours. Configured in master.conf. +# lidarr_missing_art.sh — idempotent artwork fetch. Skips existing files. # # Why separate from daily: arr libraries need to converge more than once a day. If a remote # node adds something at 2am, waiting 23 hours means both arrs search for content already owned. -# 4-hour cycles close that gap. Artwork catches up within 4 hours of a new album landing. +# 4-hour cycles close that gap. Artwork and failed imports catch up within 4 hours. # # Silent on success — runs 6x/day, only failures produce notification. # # bash /mnt/user/appdata/unraid_scripts/Orchestrators/intermediate_sync_maintenance.sh -# ── ARRS FAILED/STALLED RECOVERY ───────────────────────────────────────────────────────────── -# Schedule: 0 */6 * * * (every 6 hours) -# Background: YES -# -# Arrs never retry importFailed items automatically — this script handles it while you sleep. -# Configured arrs: Sonarr (HOST1+HOST2), Radarr (HOST1+HOST2), Lidarr (HOST1 only) -# -# Finds items with these statuses, acts on EACH: -# importFailed downloaded but couldn't import → blocklist + remove + new search -# importPending stuck in import queue → blocklist + remove + new search -# error serious failure state → blocklist + remove + new search -# stalled no seeders or no progress → blocklist + remove + new search -# -# Never touches: items with status "downloading" or "imported" -# Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (6hr default) are skipped — arr may still retry. -# -# bash /mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh - - # ── DAILY SYNC MAINTENANCE ──────────────────────────────────────────────────────────────────── # Schedule: 0 1 * * * (1am daily — runs on BOTH servers independently) # Background: YES @@ -333,6 +330,13 @@ # 8. Start local containers dependency order, new image, verify each container up # 9. docker_weekly_restart.sh restart less-critical services: NextCloud, AdGuard, Immich # 10. docker_update_remaining.sh pull latest images for all containers not in daily/weekly lists +# 11. WEEKLY_MAINTENANCE_SCRIPTS — clear_logs.sh + playback_aware_*.sh discovery scripts. +# playback_aware_lidarr_discovery.sh: 0–5 music adds/week +# based on your Emby play history + Last.fm similar artists. +# playback_aware_radarr_discovery.sh: 0–5 movie adds/week +# based on recently watched + TMDB recommendations. +# playback_aware_sonarr_discovery.sh: 0–3 TV adds/week +# based on recently watched + TMDB recommendations. # # Why weekly not nightly for Emby: HOST2 builds warm thumbnail cache throughout the week. # Nightly sync resets that cache — cold loads every morning. Weekly: cache warm 6 days, @@ -341,6 +345,29 @@ # bash /mnt/user/appdata/unraid_scripts/Orchestrators/weekly_sync_maintenance.sh +# ── MONTHLY MAINTENANCE ─────────────────────────────────────────────────────────────────────── +# Schedule: 0 3 * * * (daily 3am check — script self-gates to monthly cadence) +# Background: YES +# +# Runs heavy tasks that require a long-stable, settled system. Fires only when BOTH gates pass: +# 1. Server uptime >= MONTHLY_UPTIME_THRESHOLD_DAYS (default: 30 days) +# 2. Last run was >= MONTHLY_RUN_INTERVAL_DAYS ago (default: 30 days) +# +# Why uptime-gated not cron: a scheduled reboot resets uptime. Monthly tasks (ZFS scrub, +# SMART extended tests) need a system that has been healthy for a full month — not one that +# just rebooted. The daily 3am cron is just the trigger. The gates inside enforce the cadence. +# Silent exit 0 when either gate is not met — calling daily is always safe. +# +# Configure via master.conf MONTHLY_MAINTENANCE_SCRIPTS. Scripts planned but not yet built: +# zfs_pool_scrub.sh ZFS pool integrity scrub (Tools/) +# smart_long_test.sh SMART extended drive health test (Tools/) +# +# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh +# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh --status +# bash /mnt/user/appdata/unraid_scripts/Orchestrators/monthly_maintenance.sh --force + + # ── SUNDAY MORNING COFFEE REPORT ───────────────────────────────────────────────────────────── # Schedule: 0 7 * * 0 (Sunday 7am — after 2:30am maintenance completes) # Background: YES @@ -385,25 +412,6 @@ # bash /mnt/user/appdata/unraid_scripts/Monitors/weekly_health_digest.sh -# ── WEBGUI RESTART ──────────────────────────────────────────────────────────────────────────── -# Schedule: */10 * * * * (every 10 minutes) -# Background: YES -# -# Monitors WebGUI availability and recovers it automatically when unresponsive. -# Silent when healthy — 144 runs per day produce no output when WebGUI responds normally. -# Notification fires on any restart so you know what recovered and when. -# -# Escalation (each step only runs if previous step's recheck still fails): -# 1. curl http://localhost unresponsive? -# 2. Restart nginx wait WEBGUI_NGINX_WAIT=15s, recheck -# 3. Restart php-fpm wait WEBGUI_PHP_WAIT=10s, recheck -# handles PHP worker saturation (WebGUI feels slow) -# 4. Restart emhttp (core daemon) wait WEBGUI_EMHTTP_WAIT=30s, recheck -# 5. All failed → notify warning, manual intervention needed -# -# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/webgui_watchdog.sh - - # ── SYSTEM TUNING MONITOR ──────────────────────────────────────────────────────────────────── # Schedule: 0 */6 * * * (every 6 hours) # Background: YES @@ -423,34 +431,19 @@ # bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh -# ── CLEAR LOGS ──────────────────────────────────────────────────────────────────────────────── -# Schedule: 0 5 * * 0 (Sunday 5am — after 2:30am maintenance, before 6am monitors) -# Background: YES -# -# Clears system and Docker container logs using size thresholds. -# Only clears logs large enough to be worth clearing — preserves recent history on small logs. -# -# System logs (/var/log/syslog, messages, dmesg): only if > LOG_MIN_SIZE_MB (10MB) -# Docker container logs: only if > LOG_DOCKER_MAX_MB (100MB) per container -# Active containers (Emby, SABnzbd) grow fastest — 100MB threshold catches them. -# Truncation not rotation: > file keeps file descriptor valid for running services. -# -# bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh - - # ── SUNDAY MONITOR BLOCK ───────────────────────────────────────────────────────────────────── -# Schedule: See individual entries below — 6am through 11am Sunday -# Background: YES on all +# These monitors are now driven by sunday_morning_coffee_report.sh via COFFEE_REPORT_SCRIPTS. +# No separate User Script entries needed — configure which monitors run in master.conf. +# All monitors are read-only — observe and report, never act. Still safe to run individually. # -# Create a separate User Script entry for each monitor. Results feed into the 7am coffee report. -# All monitors are read-only — observe and report, never act. Safe to run any time. -# -# 0 6 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/zfs_memory_snapshot.sh -# 0 7 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/smart_health.sh -# 0 9 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/cert_monitor.sh -# 0 10 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/backup_verify.sh -# 0 11 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/emby_session_report.sh -# 0 11 * * 0 bash /mnt/user/appdata/unraid_scripts/Monitors/bandwidth_monitor.sh --report +# Current COFFEE_REPORT_SCRIPTS (master.conf): +# Monitors/zfs_memory_snapshot.sh ZFS pool health + ARC + Docker memory snapshot +# Monitors/smart_health.sh drive SMART attributes — reallocated, pending, temp +# Monitors/cert_monitor.sh SSL certificate expiry for all configured domains +# Monitors/backup_verify.sh rsync mirror integrity via independent MD5 checksums +# Monitors/bandwidth_monitor.sh weekly rsync transfer totals and per-share breakdown +# Monitors/emby_session_report.sh Emby usage — streams, users, library, transcode ratio +# Monitors/weekly_health_digest.sh aggregated digest — watchdogs, fallback, skip list # ============================================================================================== @@ -709,9 +702,10 @@ # UNRAID ESSENTIALS # ────────────────────────────────────────────────────────────────────────────────────────────── -# system_watchdog.sh — three-tier server stability last-resort watchdog (single-pass) -# Called every minute by watchdog_orchestrator.sh — NOT started by array_started.sh. Single-pass. -# All 18 checks independently toggleable per host in host*.conf. +# system_watchdog.sh — third layer in the watchdog chain: runs SYSTEM_WATCHDOG_SCRIPTS each cycle +# Called by watchdog_orchestrator.sh — NOT scheduled directly. Single-pass. +# Iterates SYSTEM_WATCHDOG_SCRIPTS sequentially: storage_watchdog → webgui_watchdog → network_watchdog. +# Each sub-script is independent — a failure in one does not skip the others. # # Tier 1 CRITICAL — bypass ALL strikes, reboot immediately: # Docker daemon hung → attempt rc.docker restart → still hung → reboot @@ -736,6 +730,39 @@ # bash /mnt/user/appdata/unraid_scripts/Watchdogs/stability_watchdog.sh --status # bash /mnt/user/appdata/unraid_scripts/Watchdogs/stability_watchdog.sh --dry-run +# storage_watchdog.sh — pool growth + runaway log detection (single-pass, called by system_watchdog) +# Part of SYSTEM_WATCHDOG_SCRIPTS — not scheduled directly. Called by system_watchdog.sh each cycle. +# +# Two complementary checks every cycle: +# Growth rate (zero-config): du per container vs previous baseline — any container growing +# more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan inside that dir. +# New containers covered automatically from their second cycle. No per-container config. +# Absolute log size: finds *.log files over WATCHDOG_APPDATA_LOG_MAX_GB anywhere in +# WATCHDOG_APPDATA_PATHS. Catches logs already large but not actively growing. +# +# Strike system (same pattern as docker_watchdog CPU/HTTP strikes): +# Strike 1 → warn + notify. Strike 2 → escalated notify. Strike 3 → action: +# WATCHDOG_APPDATA_TRUNCATE_LOGS=true → truncate *.log in-place, clear strikes +# WATCHDOG_APPDATA_TRUNCATE_LOGS=false → critical notify, hold strikes until resolved +# Condition resolves (growth stops / log shrinks) → strikes auto-clear. +# +# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/storage_watchdog.sh --status +# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/storage_watchdog.sh --dry-run + +# network_watchdog.sh — internet + DDNS + Tailscale + NPM proxy checks (called by system_watchdog) +# Part of SYSTEM_WATCHDOG_SCRIPTS — not scheduled directly. Called by system_watchdog.sh each cycle. +# +# Check 1 — Internet: curl external endpoint. Fail → alert + skip all remaining (prevents cascades). +# Check 2 — DDNS (Cloudflare): public IP via ifconfig.me vs DNS record via dig @1.1.1.1. +# Mismatch → restart DDNS container (triggers immediate Cloudflare record update) + notify. +# Check 3 — Tailscale: tailscale status → Running → pass. Not running → notify (no restart — human). +# Check 4 — NPM proxy (external): curl external URL through the full stack. +# 2-strike system: Strike 1 = warn + notify. Strike 2 = restart NginxProxyManager + notify. +# Strikes auto-clear when the external URL becomes reachable again. +# +# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/network_watchdog.sh --status +# bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/network_watchdog.sh --dry-run + # resource_watchdog.sh — three-level pressure reduction layer (single-pass, called by watchdog_orchestrator) # Reduces system load intelligently BEFORE docker_watchdog attempts container restarts. # Containers restarted into a resource-pressured system just fail again — this runs first. @@ -787,11 +814,15 @@ # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh --status # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/docker_syslog_filter.sh -# webgui_watchdog.sh — WebGUI availability watchdog (see ORCHESTRATORS section above) +# webgui_watchdog.sh — WebGUI availability watchdog (called by system_watchdog via SYSTEM_WATCHDOG_SCRIPTS) +# Not scheduled directly — runs as part of the every-minute watchdog chain. +# Escalation: nginx → php-fpm → emhttp, each with recheck before proceeding to next level. # bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/webgui_watchdog.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Watchdogs/System/webgui_watchdog.sh --status -# clear_logs.sh — size-threshold log cleanup (see ORCHESTRATORS section above) +# clear_logs.sh — size-threshold log cleanup (called by weekly_sync_maintenance via WEEKLY_MAINTENANCE_SCRIPTS) +# Not scheduled directly — runs as part of the Sunday 2:30am weekly window. +# System logs only cleared if > LOG_MIN_SIZE_MB (10MB). Docker logs only if > LOG_DOCKER_MAX_MB (100MB). # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/unRAID_Essentials/clear_logs.sh --status @@ -939,10 +970,40 @@ # bash /mnt/user/appdata/unraid_scripts/Media/arr_sync.sh --blocklist-remove radarr # bash /mnt/user/appdata/unraid_scripts/Media/arr_sync.sh --blocklist-list -# arrs_failed_stalled_recovery.sh — blocklist + re-search failed/stalled arr imports (see ORCHESTRATORS) +# arrs_failed_stalled_recovery.sh — blocklist + re-search failed/stalled arr imports +# Called every 4 hours by intermediate_sync_maintenance.sh via INTERMEDIATE_MAINTENANCE_SCRIPTS. +# Finds importFailed / importPending / error / stalled items across all configured arrs. +# Age gate: items newer than ARR_IMPORT_RECOVERY_AGE (6hr) are skipped — arr may still self-retry. # bash /mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Media/arrs_failed_stalled_recovery.sh --status +# playback_aware_lidarr_discovery.sh — behavior-driven music discovery (runs via WEEKLY_MAINTENANCE_SCRIPTS) +# Called by weekly_sync_maintenance.sh on Sunday. Can be run standalone. +# Stage 1: score artists you actually played this week in Emby (play count + recency + user diversity). +# Stage 2: run Last.fm artist.getSimilar on top seeds, score recommendations, add top results to Lidarr. +# Goal: 0–5 meaningful adds per run. Not bulk import — only artists likely to match your taste. +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_lidarr_discovery.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_lidarr_discovery.sh --status +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_lidarr_discovery.sh + +# playback_aware_radarr_discovery.sh — behavior-driven movie discovery (runs via WEEKLY_MAINTENANCE_SCRIPTS) +# Called by weekly_sync_maintenance.sh on Sunday. Can be run standalone. +# Stage 1: score recently watched movies in Emby, weighted by recency and TMDB rating. +# Stage 2: run TMDB recommendations on top seeds, score candidates, add top results to Radarr. +# Goal: 0–5 meaningful adds per run. +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_radarr_discovery.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_radarr_discovery.sh --status +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_radarr_discovery.sh + +# playback_aware_sonarr_discovery.sh — behavior-driven TV discovery (runs via WEEKLY_MAINTENANCE_SCRIPTS) +# Called by weekly_sync_maintenance.sh on Sunday. Can be run standalone. +# Stage 1: score recently watched series in Emby (user diversity + recency + per-user-capped episodes). +# Stage 2: run TMDB TV recommendations on top seeds, score candidates, add top results to Sonarr. +# Goal: 0–3 meaningful adds per run. +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_sonarr_discovery.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_sonarr_discovery.sh --status +# bash /mnt/user/appdata/unraid_scripts/Media/playback_aware_sonarr_discovery.sh + # lidarr_missing_art.sh — fetch missing album and artist artwork for the Lidarr music library # Called by intermediate_sync_maintenance.sh via INTERMEDIATE_MAINTENANCE_SCRIPTS. # HOST1 only — exits cleanly on HOST2 (HOST2 does not run Lidarr). @@ -1041,6 +1102,16 @@ # bash /mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh --dry-run # bash /mnt/user/appdata/unraid_scripts/Transcodes/transcode_cleanup.sh +# ramdisk_stop.sh — safely stop the transcode ramdisk without interrupting Emby +# Flips the transcode symlink to the SSD fallback BEFORE unmounting so Emby continues writing. +# Then unmounts the tmpfs and updates the state file. +# Use before: planned reboots, ramdisk size changes, ramdisk troubleshooting. +# ramdisk_setup.sh recreates and re-mounts on next array start. +# +# bash /mnt/user/appdata/unraid_scripts/Tools/ramdisk_stop.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Tools/ramdisk_stop.sh --status +# bash /mnt/user/appdata/unraid_scripts/Tools/ramdisk_stop.sh + # ────────────────────────────────────────────────────────────────────────────────────────────── # MONITORS @@ -1127,17 +1198,19 @@ # system_tuning_monitor.sh — inotify + PHP-FPM trend tracking (see ORCHESTRATORS) # bash /mnt/user/appdata/unraid_scripts/Monitors/system_tuning_monitor.sh --status -# continuous_scripts_status.sh — live dashboard for all three background processes -# The single command that shows you what the entire continuous-running ecosystem is doing. -# Shows for system_watchdog, docker_watchdog, and failover: -# Running state, PID, uptime, approximate cycle count -# Active strikes per check. Required container states. Memory-monitored container usage. -# Skip list contents (any entry needs human attention). -# Fallback state, active tiers, outage duration, remote Tailscale visibility. -# No schedule — on demand only. Run any time you want a complete status snapshot. +# mesh_monitor.sh — at-a-glance view of all servers in the mesh (read-only, on demand) +# No schedule — run any time you want a current mesh snapshot. No SSH or API calls. +# Shows each node: hostname, Tailscale IP, role (owner/mirror), services covered in fallback tiers. +# Reads directly from all three conf files — does not use detect_hosts(). # -# bash /mnt/user/appdata/unraid_scripts/Tools/continuous_scripts_status.sh -# bash /mnt/user/appdata/unraid_scripts/Tools/continuous_scripts_status.sh --log +# bash /mnt/user/appdata/unraid_scripts/Monitors/mesh_monitor.sh + +# continuous_scripts_status.sh — legacy live dashboard (Old_Arch_Still_Works/) +# NOTE: watchdog architecture changed to single-pass in v2.4. This dashboard was written +# for the continuous-loop design and is no longer kept current. Use --status flags on +# individual scripts instead: docker_watchdog --status, stability_watchdog --status, etc. +# +# bash /mnt/user/appdata/unraid_scripts/Old_Arch_Still_Works/continuous_scripts_status.sh # ────────────────────────────────────────────────────────────────────────────────────────────── @@ -1170,19 +1243,47 @@ # bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --status # bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --onboard --dry-run # bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --onboard -# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --offboard --dry-run -# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --offboard -# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --transfer --dry-run -# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh \ -# --transfer --confirm=i-understand-this-transfers-ownership # bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_manager.sh --unblock hostname +# partnership_offboard.sh — cleanly end a partnership (standalone version of --offboard) +# Role is detected automatically — run on either server. Owner path runs the full sequence: +# stop rsync → final sync → reconfigure mirror WebUIs → disable sync → local cleanup → +# Tailscale removal. Mirror path handles local side and signals owner. +# Deferred finalisation: if remote unreachable, completes automatically when reachable. +# ALWAYS --dry-run first. 10-second countdown before live execution. +# +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_offboard.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_offboard.sh --status +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_offboard.sh + +# partnership_transfer.sh — transfer ownership from current owner to current mirror +# Owner only — mirror cannot initiate. Config and WebUI targets updated; containers stay. +# Health strike system verifies both servers healthy before proceeding. +# Requires confirmation string: "i-understand-this-transfers-ownership" (long by design). +# After transfer: sync direction reverses on the next fallback/critical_sync cycle. +# +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_transfer.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_transfer.sh --status +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_transfer.sh \ +# --confirm=i-understand-this-transfers-ownership + +# gitea_ssh_setup.sh — generate Gitea SSH keypair and register it with Gitea via API +# Idempotent: skips generation if key exists, skips registration if same key already registered. +# Use --force to regenerate and re-register (e.g. after Gitea reinstall or key compromise). +# Called during onboard setup or when re-keying a server. Run after Gitea container is up. +# +# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh --status +# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh +# bash /mnt/user/appdata/unraid_scripts/Partnership/gitea_ssh_setup.sh --force + # ────────────────────────────────────────────────────────────────────────────────────────────── # INITIAL RUN # ────────────────────────────────────────────────────────────────────────────────────────────── # # Run ONCE when setting up a new server or joining a new partnership. Not scheduled. +# All initial-run scripts now live in Partnership/ alongside the lifecycle scripts they set up. # ssh_setup.sh — generate the rsync SSH keypair and install it on the remote server # Key named from hostname: unRAID-Gmer4Lfe → gmer4lfe_rsync_automation @@ -1194,11 +1295,11 @@ # At SSH_MAX_STRIKES (5): notify + return exit 2 for caller to escalate. # Strikes reset after SSH_STRIKE_RESET_HRS (24) of clean connectivity. # -# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --dry-run -# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --status -# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh -# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --force -# bash /mnt/user/appdata/unraid_scripts/Initial_run/ssh_setup.sh --validate +# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --status +# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh +# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --force +# bash /mnt/user/appdata/unraid_scripts/Partnership/ssh_setup.sh --validate # partnership_onboard.sh — full partnership setup orchestrator (run once to join a partner server) # Orchestrates the complete setup sequence in order: @@ -1213,10 +1314,10 @@ # PARTNERSHIP_OWNER_HOST correct in master.conf. # ALWAYS --dry-run first. # -# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh --dry-run -# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh -# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh --skip-ssh -# bash /mnt/user/appdata/unraid_scripts/Initial_run/partnership_onboard.sh --skip-arr-sync +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --skip-ssh +# bash /mnt/user/appdata/unraid_scripts/Partnership/partnership_onboard.sh --skip-arr-sync # ────────────────────────────────────────────────────────────────────────────────────────────── @@ -1273,6 +1374,26 @@ # bash /mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh # bash /mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh poolname +# emby_to_lidarr_sync.sh — one-shot bootstrap: add all played Emby artists to Lidarr +# Finds artists played on Emby that are not tracked in Lidarr and adds them. No scoring. +# Run once after Lidarr is set up, or any time you suspect gaps between what you listen to +# and what Lidarr monitors. Use --days N to limit lookback window. +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_lidarr_sync.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_lidarr_sync.sh --status +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_lidarr_sync.sh + +# emby_to_radarr_sync.sh — one-shot bootstrap: add all played Emby movies to Radarr +# Same pattern as emby_to_lidarr_sync. Run once to ensure everything you've watched is monitored. +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_radarr_sync.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_radarr_sync.sh --status +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_radarr_sync.sh + +# emby_to_sonarr_sync.sh — one-shot bootstrap: add all played Emby series to Sonarr +# Same pattern as emby_to_lidarr_sync. Run once to ensure all watched shows are monitored. +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_sonarr_sync.sh --dry-run +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_sonarr_sync.sh --status +# bash /mnt/user/appdata/unraid_scripts/Tools/emby_to_sonarr_sync.sh + # recreate_shares.sh — create share directories on disks after fresh install or rebuild # Use when: share .cfg files restored to fresh install but directories missing on disks. # Reads /boot/config/shares/*.cfg, parses shareInclude list, creates /mnt/diskN/sharename. @@ -1296,53 +1417,40 @@ # # * * * * * every minute: # Orchestrators/watchdog_orchestrator.sh +# └─ resource_watchdog → docker_watchdog → system_watchdog (storage/webgui/network) → stability_watchdog # # */7 * * * * every 7 minutes: # Orchestrators/transcode_management.sh # -# */10 * * * * every 10 minutes: -# Watchdogs/System/webgui_watchdog.sh -# # */30 * * * * every 30 minutes: # Orchestrators/critical_sync_maintenance.sh -# -# */30 * * * * every 30 minutes: # Rsync/rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback # # 0 */4 * * * every 4 hours: # Orchestrators/intermediate_sync_maintenance.sh +# └─ arr_sync → arrs_failed_stalled_recovery + lidarr_missing_art (INTERMEDIATE_MAINTENANCE_SCRIPTS) # # 0 */6 * * * every 6 hours: -# Media/arrs_failed_stalled_recovery.sh # Monitors/system_tuning_monitor.sh # # 0 1 * * * 1am daily: # Orchestrators/daily_sync_maintenance.sh # -# 0 5 * * 0 Sunday 5am: -# unRAID_Essentials/clear_logs.sh -# -# 0 6 * * 0 Sunday 6am: -# Monitors/zfs_memory_snapshot.sh +# 0 3 * * * 3am daily (fires monthly — self-gated by uptime + interval): +# Orchestrators/monthly_maintenance.sh # # 0 7 * * 0 Sunday 7am: # Orchestrators/sunday_morning_coffee_report.sh -# Monitors/smart_health.sh +# └─ runs all monitors via COFFEE_REPORT_SCRIPTS (master.conf): +# zfs_memory_snapshot / smart_health / cert_monitor / backup_verify +# bandwidth_monitor / emby_session_report / weekly_health_digest # # 0 8 * * * 8am daily: # Monitors/weekly_health_digest.sh # -# 0 9 * * 0 Sunday 9am: -# Monitors/cert_monitor.sh -# -# 0 10 * * 0 Sunday 10am: -# Monitors/backup_verify.sh -# -# 0 11 * * 0 Sunday 11am: -# Monitors/emby_session_report.sh -# Monitors/bandwidth_monitor.sh --report -# # 30 2 * * 0 Sunday 2:30am: # Orchestrators/weekly_sync_maintenance.sh +# └─ runs WEEKLY_MAINTENANCE_SCRIPTS after sync: +# clear_logs / playback_aware_lidarr / playback_aware_radarr / playback_aware_sonarr # # ============================================================================================== \ No newline at end of file