From 68323f9c4a48ef3cc7a35555ccca7217e08c8ade Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Mon, 1 Jun 2026 19:39:39 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20fix=20orchestrators=20README=20and=20sc?= =?UTF-8?q?ript=20headers=20=E2=80=94=2015min=20schedule,=20cron=20model,?= =?UTF-8?q?=20array=5Fstarted=20continuous=20list,=20remove=20media=5Fmana?= =?UTF-8?q?gement=20ghost=20entry,=20full=20schedule?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- Orchestrators/README-Orchestrators.md | 99 ++++++++------------------ Orchestrators/array_started.sh | 9 +-- Orchestrators/watchdog_orchestrator.sh | 8 +-- 3 files changed, 39 insertions(+), 77 deletions(-) mode change 100644 => 100755 Orchestrators/array_started.sh diff --git a/Orchestrators/README-Orchestrators.md b/Orchestrators/README-Orchestrators.md index 9aad197..92da945 100644 --- a/Orchestrators/README-Orchestrators.md +++ b/Orchestrators/README-Orchestrators.md @@ -95,7 +95,7 @@ its dependency. Everything is visible in a single summary. ## ━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -The User Scripts plugin contains exactly these entries: +The Varaverk scheduler (and User Scripts for array start) contains exactly these entries: ```bash # At Startup of Array — single entry for all startup scripts: @@ -103,10 +103,13 @@ array_started.sh # Cron — one entry per maintenance window: */7 * * * * transcode_management.sh -0 */6 * * * arrs_failed_stalled_recovery.sh -*/30 * * * * critical_sync_maintenance.sh ← auth + Emby dirty sync + partnership +*/15 * * * * watchdog_orchestrator.sh ← resource → docker → system → stability +*/30 * * * * critical_sync_maintenance.sh ← auth + Emby dirty sync + partnership +0 */4 * * * intermediate_sync_maintenance.sh ← arr sync + failed recovery + optional rsync 0 1 * * * daily_sync_maintenance.sh +0 7 * * 0 sunday_morning_coffee_report.sh 30 2 * * 0 weekly_sync_maintenance.sh +0 0 15 * * monthly_maintenance.sh ← uptime-gated: ZFS scrub, SMART tests # Manual only (not scheduled): fallback_test.sh, emby_database_repair.sh, repair tools @@ -156,7 +159,7 @@ visible. Per-item detail suppressed. phase headers, per-phase completion status, and the final summary are visible. Per-share and per-job detail suppressed. -**Daemon orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent +**High-frequency orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent during clean cycles. Only state transitions, errors, and startup-grace expiry shown without `--log`. @@ -167,13 +170,13 @@ without `--log`. | Script | What It Orchestrates | Schedule | |--------|---------------------|----------| | `array_started.sh` | All array startup scripts in order | At Startup of Array | -| `watchdog_orchestrator.sh` | resource → docker → system → stability watchdogs | Every minute | +| `watchdog_orchestrator.sh` | resource → docker → system → api_renew → stability watchdogs | Every 15 minutes | | `transcode_management.sh` | Cleanup then manager — order critical | Every 7 minutes | | `arrs_failed_stalled_recovery.sh` | Failed import + stalled download recovery | Every 6 hours | | `daily_sync_maintenance.sh` | git pull → sync → media maintenance → restarts | 1am daily | | `weekly_sync_maintenance.sh` | Stop → update → clean sync → start → weekly restarts | 2:30am Sunday | | `monthly_maintenance.sh` | Uptime-triggered heavy tasks — ZFS scrub, SMART tests | Daily check, fires when uptime ≥ 30d | -| `media_management.sh` | Permissions → cleaners → arr cleanup | Via daily_sync (or manual) | +| `intermediate_sync_maintenance.sh` | arr library sync, artwork, failed recovery | Every 4 hours | --- @@ -202,6 +205,8 @@ failed. You never need to add individual scripts to the User Scripts startup lis # ARRAY_START_SCRIPTS=( # ── One-shot scripts — run and exit naturally ───────────────────────────── + "unRAID_Essentials/unraid_api_key_renew.sh" # re-register API key FIRST — unraid-api + # registry is ephemeral, lost on service restart "unRAID_Essentials/inotify_tuning.sh" # raise inotify BEFORE containers start # containers inherit limits at startup — # if Code-Server starts with low limits @@ -210,20 +215,17 @@ ARRAY_START_SCRIPTS=( # veth interfaces — otherwise the first boot # always has unfiltered veth spam "unRAID_Essentials/php_fpm_max_children.sh" # WebGUI tuning — before any WebGUI requests - "unRAID_Essentials/ramdisk_setup.sh" # create tmpfs + symlink BEFORE Emby starts — + "Transcodes/ramdisk_setup.sh" # create tmpfs + symlink BEFORE Emby starts — # Emby needs the transcode path to exist "Docker_Essentials/docker_network_connect.sh" # ensure networks + connections BEFORE # watchdogs check container states # ── Continuous scripts — run until array stops ───────────────────────────── - "Watchdogs/stability_watchdog.sh" # last line of defense — reboots when all else fails — - # system watchdog writes state file that - # docker watchdog reads every cycle - "Watchdogs/docker_watchdog.sh" # container health BEFORE failover — - # containers must be healthy for failover - # to make reliable decisions - "Fallback/fallback.sh" # fallback LAST — needs everything else stable + "Fallback/fallback.sh" # fallback LAST — needs everything else stable ) + +# NOTE: watchdogs (docker_watchdog, system_watchdog, stability_watchdog) are NOT here. +# They run via watchdog_orchestrator.sh on cron every 15 minutes — not as daemons. ``` --- @@ -882,58 +884,6 @@ monthly_maintenance.sh --log # verbose output from each child script --- -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -## 🧹 media_management.sh -## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ - -Runs all media maintenance scripts sequentially in the order defined by -`MEDIA_MAINTENANCE_JOBS` in `master.conf`. Called by `daily_sync_maintenance.sh` -as a post-sync job — not scheduled separately. Available for manual runs when -media maintenance is needed outside the normal window. - -```bash -# Called by: daily_sync_maintenance.sh (post-sync) -# Manual use: run directly for ad hoc media maintenance -``` - ---- - -### ── Execution Order ────────────────────────────────────────────────────────── - -```bash -# master.conf -# ───────────────────────────────────────────────────────────────────────────── -# Order is critical — see README-Media.md for detailed explanation. -# Short version: permissions first, cleaners second, arr cleanup last. -# Each step depends on the previous having completed correctly. -# -MEDIA_MAINTENANCE_JOBS=( - "Media/media_shares_permissions.sh" # 1. permissions — arr cleanup depends on this - "Media/media_cleaner.sh anime" # 2. junk removal — orphan scan depends on this - "Media/media_cleaner.sh media" # 3. same for media shares - "Media/lidarr_cleanup.sh" # 4. orphan cleanup — last, after permissions + clean - "Media/sonarr_cleanup.sh" # 5. - "Media/radarr_cleanup.sh" # 6. -) -# ───────────────────────────────────────────────────────────────────────────── -# Add a script: insert in the correct position for your use case. -# Remove a script: comment it out with # -# No changes to media_management.sh needed in either case. -``` - ---- - -### ── Usage ─────────────────────────────────────────────────────────────────── - -```bash -media_management.sh # normal run — all jobs in order -media_management.sh --dry-run # preview without any deletions or changes -media_management.sh --log # verbose output from all jobs -media_management.sh --status # show configured job list and exit -``` - ---- - ## ━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ```bash @@ -943,19 +893,24 @@ media_management.sh --status # show configured job list and exit array_started.sh # ───────────────────────────────────────────────────────────────────────────── -# Frequent — every 7 minutes: +# Every 7 minutes: # ───────────────────────────────────────────────────────────────────────────── */7 * * * * transcode_management.sh +# ───────────────────────────────────────────────────────────────────────────── +# Every 15 minutes — watchdog cycle: +# ───────────────────────────────────────────────────────────────────────────── +*/15 * * * * watchdog_orchestrator.sh + # ───────────────────────────────────────────────────────────────────────────── # Every 30 minutes — auth stack + Emby dirty sync + partnership check: # ───────────────────────────────────────────────────────────────────────────── */30 * * * * critical_sync_maintenance.sh # ───────────────────────────────────────────────────────────────────────────── -# Every 6 hours: +# Every 4 hours — arr library sync + failed import recovery + optional rsync: # ───────────────────────────────────────────────────────────────────────────── -0 */6 * * * arrs_failed_stalled_recovery.sh +0 */4 * * * intermediate_sync_maintenance.sh # ───────────────────────────────────────────────────────────────────────────── # Daily — 1am: @@ -970,6 +925,12 @@ array_started.sh 50 2 * * 0 CA Auto Update plugin # plugin updates 55 2 * * 0 CA container updates # container image updates 0 3 * * 0 Network reboot # router/switch restart +0 7 * * 0 sunday_morning_coffee_report.sh + +# ───────────────────────────────────────────────────────────────────────────── +# 15th of each month (uptime-gated — silent no-op if uptime < 30 days): +# ───────────────────────────────────────────────────────────────────────────── +0 0 15 * * monthly_maintenance.sh ``` --- diff --git a/Orchestrators/array_started.sh b/Orchestrators/array_started.sh old mode 100644 new mode 100755 index 305d6fc..3554251 --- a/Orchestrators/array_started.sh +++ b/Orchestrators/array_started.sh @@ -11,6 +11,7 @@ # Current order (order matters — see below): # # ONE-SHOT (run and exit naturally): +# unRAID_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot # unRAID_Essentials/inotify_tuning.sh — raise inotify limits before containers start # unRAID_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill # unRAID_Essentials/php_fpm_max_children.sh — WebGUI performance tuning @@ -18,18 +19,18 @@ # Docker_Essentials/docker_network_connect.sh — ensure networks + container connections # # CONTINUOUS (run until array stops): -# Watchdogs/stability_watchdog.sh — system health monitor (last line of defense)) -# Watchdogs/docker_watchdog.sh — container health monitor # Fallback/fallback.sh — mutual failover monitor # +# NOTE: watchdogs (docker, system, stability) are NOT launched here. +# They run via watchdog_orchestrator.sh every 15 min (cron), not as daemons. +# # ── WHY ORDER MATTERS ───────────────────────────────────────────────────────────────────────── +# unraid_api_key_renew.sh — before anything else — self-heals API registry at boot # inotify_tuning.sh — must run BEFORE Code-Server and other containers start # containers that start with low inotify limits keep them ✅ # docker_syslog_filter — must run BEFORE any container starts creating veth interfaces # ramdisk_setup.sh — must run BEFORE Emby starts transcoding # docker_network_connect — must run BEFORE watchdogs check container states -# system_watchdog.sh — before docker_watchdog (system > container priority) -# docker_watchdog.sh — before failover (containers must be healthy for failover) # fallback.sh — last — needs everything else stable to make decisions # # ── ONE-SHOT vs CONTINUOUS DETECTION ───────────────────────────────────────────────────────── diff --git a/Orchestrators/watchdog_orchestrator.sh b/Orchestrators/watchdog_orchestrator.sh index 4f69288..6529054 100755 --- a/Orchestrators/watchdog_orchestrator.sh +++ b/Orchestrators/watchdog_orchestrator.sh @@ -7,14 +7,15 @@ # # ── EXECUTION ORDER ─────────────────────────────────────────────────────────────────────────── # Driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf — add, remove, or reorder there. -# Default: resource_watchdog → docker_watchdog → system_watchdog +# Default: resource_watchdog → docker_watchdog → system_watchdog → unraid_api_key_renew → stability_watchdog # # ── WHY ORDER MATTERS ───────────────────────────────────────────────────────────────────────── # Resource Watchdog first — frees RAM and CPU before healing attempts container restarts. # Containers restarted into a resource-pressured system just fail again. # Docker Watchdog second — restarts with pressure already reduced, more likely to stabilise. -# System Watchdog last — only triggers if prior layers could not resolve the issue. -# Rebooting without first reducing pressure may reboot into the same state. +# System Watchdog third — system component health after containers are healed. +# API key renew fourth — self-heals unraid-api registry loss; check-first, silent when valid. +# Stability Watchdog last — only reboots when all prior layers could not resolve the issue. # # ── ARRAY CHECK ─────────────────────────────────────────────────────────────────────────────── # Exits immediately if /mnt/user is not mounted as shfs (array not started). @@ -33,7 +34,6 @@ # ── REPLACES ────────────────────────────────────────────────────────────────────────────────── # Continuous loops previously in system_watchdog.sh and docker_watchdog.sh. # Those scripts are now single-pass — this orchestrator provides the cadence. -# Remove system_watchdog.sh and docker_watchdog.sh from ARRAY_START_SCRIPTS. # # ── CONFIGURATION (master.conf) ─────────────────────────────────────────────────────────────── # WATCHDOG_ORCHESTRATOR_SCRIPTS — watchdogs to run, in order