docs: fix orchestrators README and script headers — 15min schedule, cron model, array_started continuous list, remove media_management ghost entry, full schedule

Orchestrators/README-Orchestrators.md

@@ -95,7 +95,7 @@ its dependency. Everything is visible in a single summary.
 
 ## ━━━ THE ORCHESTRATOR MODEL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 
-The User Scripts plugin contains exactly these entries:
+The Varaverk scheduler (and User Scripts for array start) contains exactly these entries:
 
 ```bash
 # At Startup of Array — single entry for all startup scripts:
@@ -103,10 +103,13 @@ array_started.sh
 
 # Cron — one entry per maintenance window:
 */7   * * * *    transcode_management.sh
-0  */6 * * *     arrs_failed_stalled_recovery.sh
+*/15  * * * *    watchdog_orchestrator.sh          ← resource → docker → system → stability
-*/30  * * * *    critical_sync_maintenance.sh     ← auth + Emby dirty sync + partnership
+*/30  * * * *    critical_sync_maintenance.sh      ← auth + Emby dirty sync + partnership
+0  */4 * * *     intermediate_sync_maintenance.sh  ← arr sync + failed recovery + optional rsync
 0   1 * * *      daily_sync_maintenance.sh
+0   7 * * 0      sunday_morning_coffee_report.sh
 30  2 * * 0      weekly_sync_maintenance.sh
+0   0 15 * *     monthly_maintenance.sh            ← uptime-gated: ZFS scrub, SMART tests
 
 # Manual only (not scheduled):
 fallback_test.sh, emby_database_repair.sh, repair tools
@@ -156,7 +159,7 @@ visible. Per-item detail suppressed.
 phase headers, per-phase completion status, and the final summary are visible. Per-share
 and per-job detail suppressed.
 
-**Daemon orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
+**High-frequency orchestrators** (`watchdog_orchestrator.sh`, `transcode_management.sh`): silent
 during clean cycles. Only state transitions, errors, and startup-grace expiry shown
 without `--log`.
 
@@ -167,13 +170,13 @@ without `--log`.
 | Script | What It Orchestrates | Schedule |
 |--------|---------------------|----------|
 | `array_started.sh` | All array startup scripts in order | At Startup of Array |
-| `watchdog_orchestrator.sh` | resource → docker → system → stability watchdogs | Every minute |
+| `watchdog_orchestrator.sh` | resource → docker → system → api_renew → stability watchdogs | Every 15 minutes |
 | `transcode_management.sh` | Cleanup then manager — order critical | Every 7 minutes |
 | `arrs_failed_stalled_recovery.sh` | Failed import + stalled download recovery | Every 6 hours |
 | `daily_sync_maintenance.sh` | git pull → sync → media maintenance → restarts | 1am daily |
 | `weekly_sync_maintenance.sh` | Stop → update → clean sync → start → weekly restarts | 2:30am Sunday |
 | `monthly_maintenance.sh` | Uptime-triggered heavy tasks — ZFS scrub, SMART tests | Daily check, fires when uptime ≥ 30d |
-| `media_management.sh` | Permissions → cleaners → arr cleanup | Via daily_sync (or manual) |
+| `intermediate_sync_maintenance.sh` | arr library sync, artwork, failed recovery | Every 4 hours |
 
 ---
 
@@ -202,6 +205,8 @@ failed. You never need to add individual scripts to the User Scripts startup lis
 #
 ARRAY_START_SCRIPTS=(
     # ── One-shot scripts — run and exit naturally ─────────────────────────────
+    "unRAID_Essentials/unraid_api_key_renew.sh"   # re-register API key FIRST — unraid-api
+                                                   # registry is ephemeral, lost on service restart
     "unRAID_Essentials/inotify_tuning.sh"         # raise inotify BEFORE containers start
                                                    # containers inherit limits at startup —
                                                    # if Code-Server starts with low limits
@@ -210,20 +215,17 @@ ARRAY_START_SCRIPTS=(
                                                    # veth interfaces — otherwise the first boot
                                                    # always has unfiltered veth spam
     "unRAID_Essentials/php_fpm_max_children.sh"   # WebGUI tuning — before any WebGUI requests
-    "unRAID_Essentials/ramdisk_setup.sh"           # create tmpfs + symlink BEFORE Emby starts —
+    "Transcodes/ramdisk_setup.sh"                 # create tmpfs + symlink BEFORE Emby starts —
                                                    # Emby needs the transcode path to exist
     "Docker_Essentials/docker_network_connect.sh"  # ensure networks + connections BEFORE
                                                    # watchdogs check container states
 
     # ── Continuous scripts — run until array stops ─────────────────────────────
-    "Watchdogs/stability_watchdog.sh"         # last line of defense — reboots when all else fails —
+    "Fallback/fallback.sh"                         # fallback LAST — needs everything else stable
-                                                   # system watchdog writes state file that
-                                                   # docker watchdog reads every cycle
-    "Watchdogs/docker_watchdog.sh"         # container health BEFORE failover —
-                                                   # containers must be healthy for failover
-                                                   # to make reliable decisions
-    "Fallback/fallback.sh"                          # fallback LAST — needs everything else stable
 )
+
+# NOTE: watchdogs (docker_watchdog, system_watchdog, stability_watchdog) are NOT here.
+# They run via watchdog_orchestrator.sh on cron every 15 minutes — not as daemons.
 ```
 
 ---
@@ -882,58 +884,6 @@ monthly_maintenance.sh --log        # verbose output from each child script
 
 ---
 
-## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-## 🧹  media_management.sh
-## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-
-Runs all media maintenance scripts sequentially in the order defined by
-`MEDIA_MAINTENANCE_JOBS` in `master.conf`. Called by `daily_sync_maintenance.sh`
-as a post-sync job — not scheduled separately. Available for manual runs when
-media maintenance is needed outside the normal window.
-
-```bash
-# Called by: daily_sync_maintenance.sh (post-sync)
-# Manual use: run directly for ad hoc media maintenance
-```
-
----
-
-### ── Execution Order ──────────────────────────────────────────────────────────
-
-```bash
-# master.conf
-# ─────────────────────────────────────────────────────────────────────────────
-# Order is critical — see README-Media.md for detailed explanation.
-# Short version: permissions first, cleaners second, arr cleanup last.
-# Each step depends on the previous having completed correctly.
-#
-MEDIA_MAINTENANCE_JOBS=(
-    "Media/media_shares_permissions.sh"    # 1. permissions — arr cleanup depends on this
-    "Media/media_cleaner.sh anime"         # 2. junk removal — orphan scan depends on this
-    "Media/media_cleaner.sh media"         # 3. same for media shares
-    "Media/lidarr_cleanup.sh"             # 4. orphan cleanup — last, after permissions + clean
-    "Media/sonarr_cleanup.sh"             # 5.
-    "Media/radarr_cleanup.sh"             # 6.
-)
-# ─────────────────────────────────────────────────────────────────────────────
-# Add a script: insert in the correct position for your use case.
-# Remove a script: comment it out with #
-# No changes to media_management.sh needed in either case.
-```
-
----
-
-### ── Usage ───────────────────────────────────────────────────────────────────
-
-```bash
-media_management.sh              # normal run — all jobs in order
-media_management.sh --dry-run    # preview without any deletions or changes
-media_management.sh --log        # verbose output from all jobs
-media_management.sh --status     # show configured job list and exit
-```
-
----
-
 ## ━━━ COMPLETE SCHEDULE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 
 ```bash
@@ -943,19 +893,24 @@ media_management.sh --status     # show configured job list and exit
 array_started.sh
 
 # ─────────────────────────────────────────────────────────────────────────────
-# Frequent — every 7 minutes:
+# Every 7 minutes:
 # ─────────────────────────────────────────────────────────────────────────────
 */7 * * * *     transcode_management.sh
 
+# ─────────────────────────────────────────────────────────────────────────────
+# Every 15 minutes — watchdog cycle:
+# ─────────────────────────────────────────────────────────────────────────────
+*/15 * * * *    watchdog_orchestrator.sh
+
 # ─────────────────────────────────────────────────────────────────────────────
 # Every 30 minutes — auth stack + Emby dirty sync + partnership check:
 # ─────────────────────────────────────────────────────────────────────────────
 */30 * * * *    critical_sync_maintenance.sh
 
 # ─────────────────────────────────────────────────────────────────────────────
-# Every 6 hours:
+# Every 4 hours — arr library sync + failed import recovery + optional rsync:
 # ─────────────────────────────────────────────────────────────────────────────
-0 */6 * * *     arrs_failed_stalled_recovery.sh
+0 */4 * * *     intermediate_sync_maintenance.sh
 
 # ─────────────────────────────────────────────────────────────────────────────
 # Daily — 1am:
@@ -970,6 +925,12 @@ array_started.sh
 50 2 * * 0      CA Auto Update plugin         # plugin updates
 55 2 * * 0      CA container updates          # container image updates
 0  3 * * 0      Network reboot                # router/switch restart
+0  7 * * 0      sunday_morning_coffee_report.sh
+
+# ─────────────────────────────────────────────────────────────────────────────
+# 15th of each month (uptime-gated — silent no-op if uptime < 30 days):
+# ─────────────────────────────────────────────────────────────────────────────
+0 0 15 * *      monthly_maintenance.sh
 ```
 
 ---

Orchestrators/array_started.sh

@@ -11,6 +11,7 @@
 # Current order (order matters — see below):
 #
 #   ONE-SHOT (run and exit naturally):
+#     unRAID_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
 #     unRAID_Essentials/inotify_tuning.sh       — raise inotify limits before containers start
 #     unRAID_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
 #     unRAID_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
@@ -18,18 +19,18 @@
 #     Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
 #
 #   CONTINUOUS (run until array stops):
-#     Watchdogs/stability_watchdog.sh      — system health monitor (last line of defense))
-#     Watchdogs/docker_watchdog.sh      — container health monitor
 #     Fallback/fallback.sh                      — mutual failover monitor
 #
+#   NOTE: watchdogs (docker, system, stability) are NOT launched here.
+#   They run via watchdog_orchestrator.sh every 15 min (cron), not as daemons.
+#
 # ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
+#   unraid_api_key_renew.sh — before anything else — self-heals API registry at boot
 #   inotify_tuning.sh     — must run BEFORE Code-Server and other containers start
 #                           containers that start with low inotify limits keep them ✅
 #   docker_syslog_filter  — must run BEFORE any container starts creating veth interfaces
 #   ramdisk_setup.sh      — must run BEFORE Emby starts transcoding
 #   docker_network_connect — must run BEFORE watchdogs check container states
-#   system_watchdog.sh    — before docker_watchdog (system > container priority)
-#   docker_watchdog.sh    — before failover (containers must be healthy for failover)
 #   fallback.sh           — last — needs everything else stable to make decisions
 #
 # ── ONE-SHOT vs CONTINUOUS DETECTION ─────────────────────────────────────────────────────────

Orchestrators/watchdog_orchestrator.sh

@@ -7,14 +7,15 @@
 #
 # ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
 #   Driven by WATCHDOG_ORCHESTRATOR_SCRIPTS in master.conf — add, remove, or reorder there.
-#   Default: resource_watchdog → docker_watchdog → system_watchdog
+#   Default: resource_watchdog → docker_watchdog → system_watchdog → unraid_api_key_renew → stability_watchdog
 #
 # ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
 #   Resource Watchdog first — frees RAM and CPU before healing attempts container restarts.
 #                            Containers restarted into a resource-pressured system just fail again.
 #   Docker Watchdog second — restarts with pressure already reduced, more likely to stabilise.
-#   System Watchdog last   — only triggers if prior layers could not resolve the issue.
+#   System Watchdog third  — system component health after containers are healed.
-#                            Rebooting without first reducing pressure may reboot into the same state.
+#   API key renew fourth   — self-heals unraid-api registry loss; check-first, silent when valid.
+#   Stability Watchdog last — only reboots when all prior layers could not resolve the issue.
 #
 # ── ARRAY CHECK ───────────────────────────────────────────────────────────────────────────────
 #   Exits immediately if /mnt/user is not mounted as shfs (array not started).
@@ -33,7 +34,6 @@
 # ── REPLACES ──────────────────────────────────────────────────────────────────────────────────
 #   Continuous loops previously in system_watchdog.sh and docker_watchdog.sh.
 #   Those scripts are now single-pass — this orchestrator provides the cadence.
-#   Remove system_watchdog.sh and docker_watchdog.sh from ARRAY_START_SCRIPTS.
 #
 # ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
 #   WATCHDOG_ORCHESTRATOR_SCRIPTS         — watchdogs to run, in order