━━━━━ 📋 WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━
🖥   Identity:  HOST1 (unRAID-Gmer4Lfe)
⏱   Window:    2026-06-07 02:30:06 → 03:14:55
⏱   Duration:  44m49s
⚙   Updates:   local=true remote=true

🔄  Sync jobs (2):
  ❌ Emby
  ❌ Critical-Data
  Passed: 0  Failed: 2

⚙  Post-sync jobs:
  🏁  docker_weekly_restart.sh
  🏁  docker_update_remaining.sh
  🏁  clear_logs.sh
  🏁  playback_aware_lidarr_discovery.sh
  🏁  playback_aware_radarr_discovery.sh
  🏁  playback_aware_sonarr_discovery.sh
  🏁  docker_update.sh --remainder

⚠  [WARN]  Status: 2 failure(s)


- update-config: Use this skill to configure the Claude Code harness via settings.json. Automated behaviors ("from now on when X", "each time X", "whenever X", "before/after X") require hooks configured in settings.json - the harness executes these, not Claude, so memory/preferences cannot fulfill them. Also use for: permissions ("allow X", "add permission", "move permission to"), env vars ("set X=Y"), hook troubleshooting, or any changes to settings.json/settings.local.json files. Examples: "allow npm commands", "add bq permission to global settings", "move permission to user settings", "set DEBUG=true", "when claude stops show X". For simple settings like theme/model, suggest the /config command.
- keybindings-help: Use when the user wants to customize keyboard shortcuts, rebind keys, add chord bindings, or modify ~/.claude/keybindings.json. Examples: "rebind ctrl+s", "add a chord shortcut", "change the submit key", "customize keybindings".
- verify: Verify that a code change actually does what it's supposed to by running the app and observing behavior. Use when asked to verify a PR, confirm a fix works, test a change manually, check that a feature works, or validate local changes before pushing.
- code-review: Review the current diff for correctness bugs and reuse/simplification/efficiency cleanups at the given effort level (low/medium: fewer, high-confidence findings; high→max: broader coverage, may include uncertain findings; ultra: deep multi-agent review in the cloud). Pass --comment to post findings as inline PR comments, or --fix to apply the findings to the working tree after the review.
- simplify: Review the changed code for reuse, simplification, efficiency, and altitude cleanups, then apply the fixes. Quality only — it does not hunt for bugs; use /code-review for that.
- fewer-permission-prompts: Scan your transcripts for common read-only Bash and MCP tool calls, then add a prioritized allowlist to project .claude/settings.json to reduce permission prompts.
- loop: Run a prompt or slash command on a recurring interval (e.g. /loop 5m /foo). Omit the interval to let the model self-pace. - When the user wants to set up a recurring task, poll for status, or run something repeatedly on an interval (e.g. "check the deploy every 5 minutes", "keep running /babysit-prs"). Do NOT invoke for one-off tasks.
- schedule: Create, update, list, or run scheduled remote agents (routines) that execute on a cron schedule. - When the user wants to schedule a recurring remote agent, set up automated tasks, create a cron job for Claude Code, or manage their scheduled agents/routines. Also use when the user wants a one-time scheduled run ("run this once at 3pm", "remind me to check X tomorrow").
- claude-api: Reference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration.
TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens).
SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).
- run: Launch and drive this project's app to see a change working. Use when asked to run, start, or screenshot the app, or to confirm a change works in the real app (not just tests). First looks for a project skill that already covers launching the app; otherwise falls back to built-in patterns per project type (CLI, server, TUI, Electron, browser-driven, library).
- init: Initialize a new CLAUDE.md file with codebase documentation
- review: Review a pull request
- security-review: Complete a security review of the pending changes on the current branch

update-config keybindings-help verify code-review simplify fewer-permission-prompts loop schedule claude-api run init review security-review


Debug failed Emby and Critical-Data sync jobs


Let me look at the logs to understand why the Emby and Critical-Data syncs failed.


Bash

ls /boot/config/plugins/varaverk/data/ 2>/dev/null | head -30

List data/logs directory


List state files


cert_status.json
container_watchdog_state.db
docker_watchdog_failed.db
docker_watchdog_manual_stop.db
fallback_state.db
network_watchdog_state.db
partnership_offline_days.db
resource_watchdog_state.db
storage_watchdog_state.db
system_watchdog_oom.db
system_watchdog_reboots.db
system_watchdog_state.db
transcode_state.db
varaverk_setup.db
watchdog_appdata_growth.db
watchdog_orch_hb.count
watchdog_orchestrator.log


arr_cleanup_stats.db
arr_recovery_stats.db
bandwidth_history.db
container_restart_history.db
lidarr_discovery_history.db
lidarr_tracked.count
radarr_discovery_history.db
sonarr_discovery_history.db
transcode_daily.db
zfs-weekly-health.log


find /var/log -name "*.log" 2>/dev/null | grep -iE "weekly|emby|rsync|critical" | head -20

Find relevant log files


ls /tmp/*.log /tmp/varaverk* 2>/dev/null; find /tmp -name "*.log" 2>/dev/null | head -20

Check tmp for recent logs


/var/log/varaverk/Orchestrators/weekly_sync_maintenance.log
/var/log/varaverk/Orchestrators/critical_sync_maintenance.log


Read weekly sync maintenance log tail


━━━ 🔄 Adding to Radarr ━━━

━━━━━ 📋 DISCOVERY COMPLETE ━━━━━
  🏁  Added:    5
  ⏭  Rejected: 49
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

━━━━━ 📋 Sonarr Discovery — 2026-06-07 02:58:47 ━━━━━
🖥 HOST1 (unRAID-Gmer4Lfe) | lookback: 14d | threshold: 52/100 | max: 3

━━━ 🔄 Emby Series Library ━━━

━━━ 🔄 Emby Watch History ━━━

━━━ ⚙ Stage 1: Scoring Watched Series ━━━

━━━ 📋 Stage 1 Results (top 5 seeds) ━━━
  Seed [ 78] Dutton Ranch                             | users: 3 | vol: 8
  Seed [ 69] South Park                               | users: 2 | vol: 16
  Seed [ 63] Rick and Morty                           | users: 2 | vol: 11
  Seed [ 63] MasterChef (US)                          | users: 2 | vol: 9
  Seed [ 48] Parks and Recreation                     | users: 1 | vol: 8

━━━ 🔄 Stage 2: TMDB Recommendations ━━━

━━━ 🔄 Existing Libraries ━━━

━━━ ⚙ Stage 2: Scoring Candidates ━━━

━━━ 📋 Stage 2 Results (threshold: 52, max: 3) ━━━
  ACCEPT   [ 60] Love, Death & Robots                          seeds: 1 | rating: 8.2 | votes: 3998
  ACCEPT   [ 52] Murder Drones                                 seeds: 1 | rating: 8.5 | votes: 69
  ACCEPT   [ 52] Selena + Chef                                 seeds: 1 | rating: 8.8 | votes: 67
  REJECT   [ 47] Creature Commandos                            seeds: 1 | rating: 7.8 | votes: 327
  REJECT   [ 47] Hell on Wheels                                seeds: 1 | rating: 7.6 | votes: 463
  REJECT   [ 47] Happy Tree Friends                            seeds: 1 | rating: 7.8 | votes: 649
  REJECT   [ 44] Twilight of the Gods                          seeds: 1 | rating: 7.6 | votes: 153
  REJECT   [ 44] Voyagers!                                     seeds: 1 | rating: 7.5 | votes: 93
  REJECT   [ 44] Lazarus                                       seeds: 1 | rating: 7.7 | votes: 118
  REJECT   [ 44] Kitchen Nightmares                            seeds: 1 | rating: 7.5 | votes: 179
  REJECT   [ 40] SAINT SEIYA: Knights of the Zodiac            seeds: 1 | rating: 7.4 | votes: 668
  REJECT   [ 40] Future Man                                    seeds: 1 | rating: 7.1 | votes: 593
  REJECT   [ 37] Ransom Canyon                                 seeds: 1 | rating: 7.0 | votes: 98
  REJECT   [ 37] Lawmen: Bass Reeves                           seeds: 1 | rating: 7.0 | votes: 167
  REJECT   [ 33] Killjoys                                      seeds: 1 | rating: 6.9 | votes: 521
  REJECT   [ 33] Squid Game: The Challenge                     seeds: 1 | rating: 6.5 | votes: 630
  REJECT   [ 30] So You Think You Can Dance                    seeds: 1 | rating: 6.8 | votes: 76
  REJECT   [ 30] The Amazing Race                              seeds: 1 | rating: 6.9 | votes: 164
  REJECT   [ 30] Sausage Party: Foodtopia                      seeds: 1 | rating: 6.8 | votes: 199
  REJECT   [ 30] The Mole                                      seeds: 1 | rating: 6.9 | votes: 61
  REJECT   [ 30] Marvel's M.O.D.O.K.                           seeds: 1 | rating: 6.6 | votes: 195

  Already in library: 27 | Below quality floor: 34 | Cooldown: 16
  ACCEPT: 3 | REJECT: 18

━━━ 🔄 Adding to Sonarr ━━━

━━━━━ 📋 DISCOVERY COMPLETE ━━━━━
  🏁  Added:    3
  ⏭  Rejected: 18
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

━━━ 📦 Remainder Container Updates ━━━

━━━ 📦 Docker Update (remainder) — 2026-06-07 02:59:29 ━━━
📦 Updating 47 container(s) (not in daily or weekly sync)

━━━━━ 📋 DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━
🖥       Identity:    HOST1 (unRAID-Gmer4Lfe)
⏱       Duration:    15m26s
🏁       Updated:     1
🔄       Rebuilt:     1
🔄       Pruned:      Total reclaimed space: 6.303GB
🏁 Status: done ✅ — 1 updated, 46 current
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Remainder updates complete ✅

━━━━━ 📋 WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━
🖥   Identity:  HOST1 (unRAID-Gmer4Lfe)
⏱   Window:    2026-06-07 02:30:06 → 03:14:55
⏱   Duration:  44m49s
⚙   Updates:   local=true remote=true

🔄  Sync jobs (2):
  ❌ Emby
  ❌ Critical-Data
  Passed: 0  Failed: 2

⚙  Post-sync jobs:
  🏁  docker_weekly_restart.sh
  🏁  docker_update_remaining.sh
  🏁  clear_logs.sh
  🏁  playback_aware_lidarr_discovery.sh
  🏁  playback_aware_radarr_discovery.sh
  🏁  playback_aware_sonarr_discovery.sh
  🏁  docker_update.sh --remainder

⚠  [WARN]  Status: 2 failure(s)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━


━━━━━ 📋 WEEKLY SYNC MAINTENANCE SUMMARY ━━━━━ 🖥   Identity:  HOST1 (unRAID-Gmer4Lfe) ⏱   Window:    2026-06-07 02:30:06 → 03:14:55 ⏱   Duration:  44m49s ⚙   Updates:   local=true remote=true  🔄…


The sync failures are logged earlier in the file. Let me find them.


grep -n "FAIL\|ERROR\|error\|failed\|rsync\|Emby\|Critical-Data\|❌\|exit\|Exit" /var/log/varaverk/Orchestrators/weekly_sync_maintenance.log | head -60

Find failure lines in weekly log


24:━━━ [1/2] Emby ━━━
25:⚠  [WARN]  rsync has been running for 36213s — may be stuck (PID 2486476)
26:❌ [ERROR] Another instance of rsync is already running (PID 2486476) — exiting
27:❌ [ERROR] Emby — failed after 0s (exit 1)
30:━━━ [2/2] Critical-Data ━━━
31:⚠  [WARN]  rsync has been running for 36213s — may be stuck (PID 2486476)
32:❌ [ERROR] Another instance of rsync is already running (PID 2486476) — exiting
33:❌ [ERROR] Critical-Data — failed after 0s (exit 1)
85:━━━ 🔄 Emby Play History ━━━
122:━━━ 🔄 Emby Watch History ━━━
209:━━━ 🔄 Emby Series Library ━━━
211:━━━ 🔄 Emby Watch History ━━━
283:  ❌ Emby
284:  ❌ Critical-Data


Root cause is clear: a stuck rsync (PID 2486476) had been running for 36,213 seconds (~10 hours) and blocked both syncs. Let me check if it's still alive.


ps -p 2486476 -o pid,ppid,etime,cmd 2>/dev/null || echo "Process not found (already dead)"

Check if stuck rsync PID is still alive


List all running rsync processes


    PID    PPID     ELAPSED CMD
2486476 2481818    16:51:01 bash /boot/config/plugins/varaverk/Orchestrators/../Rsync/rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-fallback


root     2486476  0.0  0.0   6904  5464 ?        S    Jun06   0:02 bash /boot/config/plugins/varaverk/Orchestrators/../Rsync/rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-fallback
root     2488605  0.0  0.0   2716  1708 ?        S    Jun06   0:00 tee -a /tmp/unraid_locks/rsync_critical-fallback.log


It's still alive — 16 hours 51 minutes. Let me see what it's actually doing and whether it has a child rsync process.


pstree -p 2486476 2>/dev/null || ps --ppid 2486476 -o pid,etime,cmd 2>/dev/null

Show process tree for stuck rsync wrapper


Read the stuck rsync log tail


━━━ 🛡 Pre-flight ━━━
ℹ  [INFO]  📡 unRAID-Jayred365 is reachable
ℹ  [INFO]  🩺 Remote rootfs: 1% (threshold: 75%)
ℹ  [INFO]  🩺 Remote share verified: /mnt/user/appdata-Fallback/Critical-Data
ℹ  [INFO]  💾 Checking disks backing Critical-Data on unRAID-Jayred365...
ℹ  [INFO]  💾 docker-unraid (ZFS pool) 🟢 — Critical-Data ONLINE
🏁  [OK]    All disks backing Critical-Data are online ✅

━━━ 🔄 Transfer ━━━
🚀  Source:   /mnt/user/appdata-Fallback/Critical-Data
🌐  Remote:   100.97.4.47:/mnt/user/appdata-Fallback/Critical-Data
⚙ Profile:  critical-fallback
🖥 Identity: HOST1 → HOST2

🔄  Rsync running — this may take a while...

━━━ ▶ 📦 Remote Restart (post dirty sync) ━━━
📶 Logged: critical-fallback — 9s — success

━━━━━ 📋 RSYNC SUMMARY ━━━━━
🖥    Identity:  HOST1 (unRAID-Gmer4Lfe)
🚀     Directory: /mnt/user/appdata-Fallback/Critical-Data
⚙    Profile:   critical-fallback
⏱    Duration:  9s
🏁 Status: ✅ DONE
━━━━━━━━━━━━━━━━━━━━━━━


Interesting — the rsync log says it finished successfully in 9s, but the bash wrapper is still alive 16+ hours later. Let me check the lock mechanism and what's keeping the process alive.


List lock files


List open file descriptors on stuck bash process


total 24
drwxr-xr-x  2 root root  160 Jun  7 09:21 ./
drwxrwxrwt 18 root root 1220 Jun  7 09:21 ../
-rw-r--r--  1 root root   34 Jun  6 16:30 critical_sync_maintenance.lock
-rw-r--r--  1 root root   15 Jun  5 15:36 fallback.lock
-rw-r--r--  1 root root   14 Jun  6 16:30 rsync.lock
-rw-r--r--  1 root root    2 Jun  6 16:30 rsync_active_count
-rw-r--r--  1 root root   32 Jun  6 16:30 rsync_critical-fallback.lock
-rw-r--r--  1 root root 1160 Jun  6 16:30 rsync_critical-fallback.log


COMMAND     PID USER  FD   TYPE DEVICE SIZE/OFF     NODE NAME
bash    2486476 root cwd    DIR    0,2      320        4 /root
bash    2486476 root rtd    DIR    0,2      440        1 /
bash    2486476 root txt    REG    0,2  1362000      517 /bin/bash
bash    2486476 root mem    REG    0,2  2420432     3751 /lib64/libc-2.43.so
bash    2486476 root mem    REG    0,2   213592     3821 /lib64/libtinfo.so.6.6
bash    2486476 root mem    REG    0,2   270488     3745 /lib64/ld-linux-x86-64-2.43.so
bash    2486476 root   0u   CHR    1,3      0t0        4 /dev/null
bash    2486476 root   2w  FIFO   0,14      0t0 91682694 pipe
bash    2486476 root 255r   REG   0,25    20114    13336 /boot/config/plugins/varaverk/Rsync/rsync.sh


cat /tmp/unraid_locks/rsync.lock /tmp/unraid_locks/rsync_critical-fallback.lock /tmp/unraid_locks/rsync_active_count 2>/dev/null | cat -v

Read lock file contents


2486476:rsync
2486476:rsync_critical-fallback
1


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Rsync Core Script ==============================================
4	# ==============================================================================================
5	#
6	# PURPOSE
7	# ─────────────────────────────────────────────────────────────────────────────
8	# Core rsync engine for the two-server ecosystem. Called per share or per
9	# appdata profile by orchestrators (daily_sync_maintenance, weekly_sync_maintenance,
10	# critical_sync_maintenance) and directly for manual or scheduled dirty syncs.
11	#
12	# Profile is inferred from the directory basename (lowercased). Override with
13	# --profile=name for explicit selection. If no profile matches, global defaults
14	# from master.conf apply and no containers are stopped.
15	#
16	# After each sync, logs transfer data to bandwidth_monitor.sh for the weekly
17	# bandwidth report. Silent on success — only failures produce visible output.
18	#
19	# ==============================================================================================
20	# OPERATIONAL MODEL
21	# ==============================================================================================
22	#
23	# Profiles define per-share behavior:
24	#   PROFILE_CRITICAL_CONTAINER_NAMES  — containers stopped on both servers before sync
25	#   PROFILE_DELAYED_CONTAINERS        — containers with a delay before restart after sync
26	#   PROFILE_CONTAINER_DELAY           — seconds before delayed containers start
27	#   PROFILE_RSYNC_OPTS                — rsync flags (does not inherit DEFAULT_RSYNC_OPTS)
28	#   PROFILE_BW_LIMIT                  — bandwidth limit in KB/s
29	#   PROFILE_RETRY_COUNT               — retry attempts on failure
30	#   PROFILE_SLEEP                     — seconds between retry attempts
31	#   PROFILE_EXCLUDE_DIRS              — paths excluded from transfer
32	#   PROFILE_REMOTE_RESTART_CONTAINERS — containers restarted on remote after dirty sync
33	#                                       Was running → restart. Was stopped → leave stopped.
34	#
35	# Two-tier rsync enable/disable:
36	#   Tier 1: RSYNC_ENABLED=false → all rsync stops immediately (checked by this script)
37	#   Tier 2: per-orchestrator flag (DAILY_RSYNC_ENABLED etc.) → checked by caller
38	#
39	# Bandwidth logging: after each sync, logs profile/duration/status/bytes to
40	# bandwidth_monitor.sh --log-transfer. Bytes captured from rsync --stats via awk
41	# using version-stable field names.
42	#
43	# ==============================================================================================
44	# OPERATIONAL SAFEGUARDS
45	# ==============================================================================================
46	#
47	# Global Rsync Gate
48	#   check_rsync_enabled() — RSYNC_ENABLED=false exits cleanly before any operation.
49	#
50	# Partnership Blocklist
51	#   Refuses to sync if REMOTE_SERVER_NAME appears in the partnership blocklist.
52	#   Written at offboard — prevents stale access after a partnership ends.
53	#
54	# Version Parity
55	#   check_unraid_version_parity — refuses sync if servers on incompatible unRAID versions.
56	#
57	# Remote Health Pre-flights
58	#   check_connectivity() — Tailscale IP reachable before any SSH
59	#   check_remote_rootfs() — aborts if remote rootfs exceeds ROOTFS_WARN_PCT
60	#   check_remote_share() — aborts if target directory missing or empty on remote
61	#   check_remote_disks() — verifies all backing disks online on remote
62	#
63	# Drive Temperature Check
64	#   check_local_disk_temps() — runs before any transfer. Exit 1 = skip this profile,
65	#   exit 2 = abort all remaining syncs (CRITICAL temperature).
66	#
67	# Remote Docker Daemon Check
68	#   check_remote_docker_daemon — verified before any container stop/start operations.
69	#   If daemon unresponsive: container operations skipped, rsync proceeds without stopping.
70	#
71	# Per-Profile Concurrency Lock
72	#   acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.
73	#   Global concurrent limit prevents too many simultaneous rsync processes.
74	#
75	# Notification Validated
76	#   platform_require_cmd confirms the notify script is present before use.
77	#
78	# ==============================================================================================
79	# CONFIGURATION
80	# ==============================================================================================

#!/bin/bash
# ==============================================================================================
# ============================= Rsync Core Script ==============================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# Core rsync engine for the two-server ecosystem. Called per share or per
# appdata profile by orchestrators (daily_sync_maintenance, weekly_sync_maintenance,
# critical_sync_maintenance) and directly for manual or scheduled dirty syncs.
#
# Profile is inferred from the directory basename (lowercased). Override with
# --profile=name for explicit selection. If no profile matches, global defaults
# from master.conf apply and no containers are stopped.
#
# After each sync, logs transfer data to bandwidth_monitor.sh for the weekly
# bandwidth report. Silent on success — only failures produce visible output.
#
# ==============================================================================================
# OPERATIONAL MODEL
# ==============================================================================================
#
# Profiles define per-share behavior:
#   PROFILE_CRITICAL_CONTAINER_NAMES  — containers stopped on both servers before sync
#   PROFILE_DELAYED_CONTAINERS        — containers with a delay before restart after sync
#   PROFILE_CONTAINER_DELAY           — seconds before delayed containers start
#   PROFILE_RSYNC_OPTS                — rsync flags (does not inherit DEFAULT_RSYNC_OPTS)
#   PROFILE_BW_LIMIT                  — bandwidth limit in KB/s
#   PROFILE_RETRY_COUNT               — retry attempts on failure
#   PROFILE_SLEEP                     — seconds between retry attempts
#   PROFILE_EXCLUDE_DIRS              — paths excluded from transfer
#   PROFILE_REMOTE_RESTART_CONTAINERS — containers restarted on remote after dirty sync
#                                       Was running → restart. Was stopped → leave stopped.
#
# Two-tier rsync enable/disable:
#   Tier 1: RSYNC_ENABLED=false → all rsync stops immediately (checked by this script)
#   Tier 2: per-orchestrator flag (DAILY_RSYNC_ENABLED etc.) → checked by caller
#
# Bandwidth logging: after each sync, logs profile/duration/status/bytes to
# bandwidth_monitor.sh --log-transfer. Bytes captured from rsync --stats via awk
# using version-stable field names.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Global Rsync Gate
#   check_rsync_enabled() — RSYNC_ENABLED=false exits cleanly before any operation.
#
# Partnership Blocklist
#   Refuses to sync if REMOTE_SERVER_NAME appears in the partnership blocklist.
#   Written at offboard — prevents stale access after a partnership ends.
#
# Version Parity
#   check_unraid_version_parity — refuses sync if servers on incompatible unRAID versions.
#
# Remote Health Pre-flights
#   check_connectivity() — Tailscale IP reachable before any SSH
#   check_remote_rootfs() — aborts if remote rootfs exceeds ROOTFS_WARN_PCT
#   check_remote_share() — aborts if target directory missing or empty on remote
#   check_remote_disks() — verifies all backing disks online on remote
#
# Drive Temperature Check
#   check_local_disk_temps() — runs before any transfer. Exit 1 = skip this profile,
#   exit 2 = abort all remaining syncs (CRITICAL temperature).
#
# Remote Docker Daemon Check
#   check_remote_docker_daemon — verified before any container stop/start operations.
#   If daemon unresponsive: container operations skipped, rsync proceeds without stopping.
#
# Per-Profile Concurrency Lock
#   acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.
#   Global concurrent limit prevents too many simultaneous rsync processes.
#
# Notification Validated
#   platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# CONFIGURATION
# ==============================================================================================


grep -n "acquire_rsync_lock\|release_rsync_lock\|rsync_lock\|active_count\|wait\|trap\|EXIT\|cleanup" /boot/config/plugins/varaverk/Rsync/rsync.sh | head -40

Find lock and cleanup logic in rsync.sh


72:#   acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.
89:#       media shares spread files only, arr cleanup scripts own deletions. Profile-specific
206:acquire_rsync_lock "$PROFILE_NAME"
330:    RSYNC_EXIT=$?
332:    if [[ "$RSYNC_EXIT" -eq 0 ]]; then
343:        log "Exit code: $RSYNC_EXIT"
456:exec 1>&-; wait


445	    warn "DRY RUN — no changes made"
446	elif [[ "$RSYNC_SUCCESS" == true ]]; then
447	    echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
448	else
449	    echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
450	    notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
451	        "Rsync" "warning"
452	fi
453	echo "━━━━━━━━━━━━━━━━━━━━━━━"
454
455	# Flush tee and preserve log for UI
456	exec 1>&-; wait
457	cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
458	rm -f "$VV_LIVE_LOG"
459
460	[[ "$RSYNC_SUCCESS" == false ]] && [[ "$DRY_RUN" == false ]] && exit 1
461	exit 0

    warn "DRY RUN — no changes made"
elif [[ "$RSYNC_SUCCESS" == true ]]; then
    echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
else
    echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
    notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
        "Rsync" "warning"
fi
echo "━━━━━━━━━━━━━━━━━━━━━━━"

# Flush tee and preserve log for UI
exec 1>&-; wait
cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
rm -f "$VV_LIVE_LOG"

[[ "$RSYNC_SUCCESS" == false ]] && [[ "$DRY_RUN" == false ]] && exit 1
exit 0


grep -n "acquire_rsync_lock\|release_rsync_lock" /boot/config/plugins/varaverk/common.sh | head -20

Find lock functions in common.sh


19:#   Locking          — acquire_lock, acquire_rsync_lock — concurrent execution protection
47:#            Script locking system added — acquire_lock(), acquire_rsync_lock()
1351:# Prevents the single-trap-per-acquire problem: each new acquire_lock/acquire_rsync_lock
1462:# Usage: acquire_rsync_lock "$PROFILE_NAME"
1463:acquire_rsync_lock() {


1460	# Acquire per-profile rsync lock + enforce global concurrent limit.
1461	# Prevents same profile running twice and limits total concurrent rsync instances.
1462	# Usage: acquire_rsync_lock "$PROFILE_NAME"
1463	acquire_rsync_lock() {
1464	    local profile="$1"
1465	    local profile_lock
1466	    profile_lock="$(_lock_file "rsync_${profile}")"
1467
1468	    mkdir -p "$LOCK_DIR"
1469
1470	    # Per-profile lock — same profile cannot run twice
1471	    if [[ -f "$profile_lock" ]]; then
1472	        local lock_content existing_pid locked_name
1473	        lock_content=$(cat "$profile_lock" 2>/dev/null)
1474	        existing_pid="${lock_content%%:*}"
1475	        locked_name="${lock_content##*:}"
1476	        if [[ -n "$existing_pid" ]] && kill -0 "$existing_pid" 2>/dev/null && \
1477	           [[ "$locked_name" == "rsync_${profile}" ]]; then
1478	            error "rsync profile '$profile' is already running (PID $existing_pid) — exiting"
1479	            exit 1
1480	        else
1481	            warn "Stale rsync lock for profile '$profile' — clearing"
1482	            rm -f "$profile_lock"
1483	        fi
1484	    fi
1485
1486	    # Global concurrent limit — validate count against actual live locks
1487	    local current_count=0
1488	    if [[ -f "$RSYNC_COUNT_FILE" ]]; then
1489	        current_count=$(cat "$RSYNC_COUNT_FILE" 2>/dev/null || echo 0)
1490	        local actual_count=0
1491	        for lf in "$LOCK_DIR"/rsync_*.lock; do
1492	            [[ -f "$lf" ]] || continue
1493	            local lpid lf_content
1494	            lf_content=$(cat "$lf" 2>/dev/null)
1495	            lpid="${lf_content%%:*}"
1496	            kill -0 "$lpid" 2>/dev/null && ((actual_count++))
1497	        done
1498	        if [[ "$actual_count" -ne "$current_count" ]]; then
1499	            log "rsync count corrected: $current_count → $actual_count"
1500	            current_count=$actual_count
1501	            echo "$current_count" > "$RSYNC_COUNT_FILE"
1502	        fi
1503	    fi
1504
1505	    if [[ "$current_count" -ge "$RSYNC_MAX_CONCURRENT" ]]; then
1506	        error "Maximum concurrent rsync limit ($RSYNC_MAX_CONCURRENT) reached — exiting"
1507	        info "Active rsync locks: $(ls "$LOCK_DIR"/rsync_*.lock 2>/dev/null | \
1508	            xargs -I{} basename {} .lock | tr '\n' ' ')"
1509	        exit 1
1510	    fi
1511
1512	    # Acquire profile lock and increment counter
1513	    echo "$$:rsync_${profile}" > "$profile_lock"
1514	    echo $(( current_count + 1 )) > "$RSYNC_COUNT_FILE"
1515	    _register_lock "$profile_lock"
1516	    log "$ICON_LOCK rsync lock acquired: profile '$profile' (PID $$, active: $(( current_count + 1 ))/$RSYNC_MAX_CONCURRENT)"
1517	}
1518
1519	# Internal — release rsync lock and decrement global counter on exit

# Acquire per-profile rsync lock + enforce global concurrent limit.
# Prevents same profile running twice and limits total concurrent rsync instances.
# Usage: acquire_rsync_lock "$PROFILE_NAME"
acquire_rsync_lock() {
    local profile="$1"
    local profile_lock
    profile_lock="$(_lock_file "rsync_${profile}")"

    mkdir -p "$LOCK_DIR"

    # Per-profile lock — same profile cannot run twice
    if [[ -f "$profile_lock" ]]; then
        local lock_content existing_pid locked_name
        lock_content=$(cat "$profile_lock" 2>/dev/null)
        existing_pid="${lock_content%%:*}"
        locked_name="${lock_content##*:}"
        if [[ -n "$existing_pid" ]] && kill -0 "$existing_pid" 2>/dev/null && \
           [[ "$locked_name" == "rsync_${profile}" ]]; then
            error "rsync profile '$profile' is already running (PID $existing_pid) — exiting"
            exit 1
        else
            warn "Stale rsync lock for profile '$profile' — clearing"
            rm -f "$profile_lock"
        fi
    fi

    # Global concurrent limit — validate count against actual live locks
    local current_count=0
    if [[ -f "$RSYNC_COUNT_FILE" ]]; then
        current_count=$(cat "$RSYNC_COUNT_FILE" 2>/dev/null || echo 0)
        local actual_count=0
        for lf in "$LOCK_DIR"/rsync_*.lock; do
            [[ -f "$lf" ]] || continue
            local lpid lf_content
            lf_content=$(cat "$lf" 2>/dev/null)
            lpid="${lf_content%%:*}"
            kill -0 "$lpid" 2>/dev/null && ((actual_count++))
        done
        if [[ "$actual_count" -ne "$current_count" ]]; then
            log "rsync count corrected: $current_count → $actual_count"
            current_count=$actual_count
            echo "$current_count" > "$RSYNC_COUNT_FILE"
        fi
    fi

    if [[ "$current_count" -ge "$RSYNC_MAX_CONCURRENT" ]]; then
        error "Maximum concurrent rsync limit ($RSYNC_MAX_CONCURRENT) reached — exiting"
        info "Active rsync locks: $(ls "$LOCK_DIR"/rsync_*.lock 2>/dev/null | \
            xargs -I{} basename {} .lock | tr '\n' ' ')"
        exit 1
    fi

    # Acquire profile lock and increment counter
    echo "$$:rsync_${profile}" > "$profile_lock"
    echo $(( current_count + 1 )) > "$RSYNC_COUNT_FILE"
    _register_lock "$profile_lock"
    log "$ICON_LOCK rsync lock acquired: profile '$profile' (PID $$, active: $(( current_count + 1 ))/$RSYNC_MAX_CONCURRENT)"
}

# Internal — release rsync lock and decrement global counter on exit


1518
1519	# Internal — release rsync lock and decrement global counter on exit
1520	_release_rsync_on_exit() {
1521	    local profile_lock="$1"
1522	    [[ -f "$profile_lock" ]] && rm -f "$profile_lock"
1523	    if [[ -f "$RSYNC_COUNT_FILE" ]]; then
1524	        local count
1525	        count=$(cat "$RSYNC_COUNT_FILE" 2>/dev/null || echo 1)
1526	        count=$(( count - 1 ))
1527	        [[ "$count" -lt 0 ]] && count=0
1528	        echo "$count" > "$RSYNC_COUNT_FILE"
1529	    fi
1530	}
1531
1532	# ==============================================================================================
1533	# ── PATH TRANSLATION ──────────────────────────────────────────────────────────────────────────
1534	# ==============================================================================================
1535	# Translates container-internal file paths to host paths for arr cleanup scripts.
1536	# Arr APIs return file paths as seen inside the container — scripts need host paths to scan.
1537	#
1538	# Uses ARR_PATH_MAP associative array — must be declared before calling.
1539	# Longest-match wins — prevents partial path collisions.
1540	#
1541	# Usage:
1542	#   declare -A ARR_PATH_MAP=(["/ext-music"]="/mnt/user/Music-New")
1543	#   translate_path "/ext-music/Artist/Album/track.flac"
1544	#   Returns: /mnt/user/Music-New/Artist/Album/track.flac
1545	#
1546	# If no match found — returns path unchanged (safe fallback).
1547

# Internal — release rsync lock and decrement global counter on exit
_release_rsync_on_exit() {
    local profile_lock="$1"
    [[ -f "$profile_lock" ]] && rm -f "$profile_lock"
    if [[ -f "$RSYNC_COUNT_FILE" ]]; then
        local count
        count=$(cat "$RSYNC_COUNT_FILE" 2>/dev/null || echo 1)
        count=$(( count - 1 ))
        [[ "$count" -lt 0 ]] && count=0
        echo "$count" > "$RSYNC_COUNT_FILE"
    fi
}

# ==============================================================================================
# ── PATH TRANSLATION ──────────────────────────────────────────────────────────────────────────
# ==============================================================================================
# Translates container-internal file paths to host paths for arr cleanup scripts.
# Arr APIs return file paths as seen inside the container — scripts need host paths to scan.
#
# Uses ARR_PATH_MAP associative array — must be declared before calling.
# Longest-match wins — prevents partial path collisions.
#
# Usage:
#   declare -A ARR_PATH_MAP=(["/ext-music"]="/mnt/user/Music-New")
#   translate_path "/ext-music/Artist/Album/track.flac"
#   Returns: /mnt/user/Music-New/Artist/Album/track.flac
#
# If no match found — returns path unchanged (safe fallback).


80	# ==============================================================================================
81	#
82	# master.conf
83	#
84	#   RSYNC_ENABLED
85	#       Global on/off toggle for all rsync operations. (default: true)
86	#
87	#   DEFAULT_RSYNC_OPTS
88	#       Base rsync flags for unproiled shares. Note: --delete is intentionally absent —
89	#       media shares spread files only, arr cleanup scripts own deletions. Profile-specific
90	#       opts set --delete explicitly where needed.
91	#
92	#   BW_LIMIT
93	#       Default bandwidth cap in KB/s when no PROFILE_BW_LIMIT is set. (default: 0 = unlimited)
94	#
95	#   RETRY_COUNT
96	#       Default retry attempts on rsync failure. (default: 3)
97	#
98	#   SLEEP
99	#       Default seconds between retry attempts. (default: 60)
100	#
101	#   ROOTFS_WARN_PCT
102	#       Abort threshold for remote rootfs percentage full. (default: 75)
103	#
104	#   PROFILES["profile_KEY"]
105	#       Profile definitions — one entry per PROFILE_* key per profile name.
106	#       See OPERATIONAL MODEL above for all supported keys.
107	#
108	#   BANDWIDTH_LOG / BANDWIDTH_WARN_GB
109	#       Shared with bandwidth_monitor.sh — set once, used by both.
110	#
111	# ==============================================================================================
112	# RUNTIME MODES
113	# ==============================================================================================
114	#
115	# rsync.sh /path/to/share
116	#     Sync the given path to the remote. Profile inferred from directory basename.
117	#
118	# rsync.sh /path/to/share --profile=name
119	#     Sync with explicit profile override — bypasses basename inference.
120	#
121	# rsync.sh /path/to/share --dry-run
122	#     Run all pre-flight checks and show what rsync would transfer. No transfer,
123	#     no container stops.
124	#
125	# rsync.sh /path/to/share --status
126	#     Show resolved profile, remote identity, and configuration. Then exit.
127	#
128	# rsync.sh /path/to/share --log
129	#     Verbose output throughout — every decision logged.
130	#
131	# ==============================================================================================
132
133	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
134
135	source "$SCRIPT_DIR/../load_config.sh"
136
137	# ── Separate positional directory arg from flags ───────────────────────────────────────────────
138	DIRECTORY=""
139	PROFILE_OVERRIDE=""
140	RAW_ARGS=()
141
142	for ARG in "$@"; do
143	    case "$ARG" in
144	        --profile=*) PROFILE_OVERRIDE="${ARG#--profile=}" ;;
145	        --*|*=*)     RAW_ARGS+=("$ARG") ;;
146	        *)           [[ -z "$DIRECTORY" ]] && DIRECTORY="$ARG" || RAW_ARGS+=("$ARG") ;;
147	    esac
148	done
149
150	parse_args "${RAW_ARGS[@]}"
151
152	[[ -z "$DIRECTORY" ]] && {
153	    error "No directory specified"
154	    error "Usage: rsync.sh <dir> [--dry-run] [--log] [--profile=name]"
155	    exit 1
156	}
157
158	# ==============================================================================================
159	# ━━━ Setup ━━━

# ==============================================================================================
#
# master.conf
#
#   RSYNC_ENABLED
#       Global on/off toggle for all rsync operations. (default: true)
#
#   DEFAULT_RSYNC_OPTS
#       Base rsync flags for unproiled shares. Note: --delete is intentionally absent —
#       media shares spread files only, arr cleanup scripts own deletions. Profile-specific
#       opts set --delete explicitly where needed.
#
#   BW_LIMIT
#       Default bandwidth cap in KB/s when no PROFILE_BW_LIMIT is set. (default: 0 = unlimited)
#
#   RETRY_COUNT
#       Default retry attempts on rsync failure. (default: 3)
#
#   SLEEP
#       Default seconds between retry attempts. (default: 60)
#
#   ROOTFS_WARN_PCT
#       Abort threshold for remote rootfs percentage full. (default: 75)
#
#   PROFILES["profile_KEY"]
#       Profile definitions — one entry per PROFILE_* key per profile name.
#       See OPERATIONAL MODEL above for all supported keys.
#
#   BANDWIDTH_LOG / BANDWIDTH_WARN_GB
#       Shared with bandwidth_monitor.sh — set once, used by both.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# rsync.sh /path/to/share
#     Sync the given path to the remote. Profile inferred from directory basename.
#
# rsync.sh /path/to/share --profile=name
#     Sync with explicit profile override — bypasses basename inference.
#
# rsync.sh /path/to/share --dry-run
#     Run all pre-flight checks and show what rsync would transfer. No transfer,
#     no container stops.
#
# rsync.sh /path/to/share --status
#     Show resolved profile, remote identity, and configuration. Then exit.
#
# rsync.sh /path/to/share --log
#     Verbose output throughout — every decision logged.
#
# ==============================================================================================

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

source "$SCRIPT_DIR/../load_config.sh"

# ── Separate positional directory arg from flags ───────────────────────────────────────────────
DIRECTORY=""
PROFILE_OVERRIDE=""
RAW_ARGS=()

for ARG in "$@"; do
    case "$ARG" in
        --profile=*) PROFILE_OVERRIDE="${ARG#--profile=}" ;;
        --*|*=*)     RAW_ARGS+=("$ARG") ;;
        *)           [[ -z "$DIRECTORY" ]] && DIRECTORY="$ARG" || RAW_ARGS+=("$ARG") ;;
    esac
done

parse_args "${RAW_ARGS[@]}"

[[ -z "$DIRECTORY" ]] && {
    error "No directory specified"
    error "Usage: rsync.sh <dir> [--dry-run] [--log] [--profile=name]"
    exit 1
}

# ==============================================================================================
# ━━━ Setup ━━━


158	# ==============================================================================================
159	# ━━━ Setup ━━━
160	# ==============================================================================================
161	if [[ "$EUID" -ne 0 ]]; then
162	    error "Must be run as root"
163	    exit 1
164	fi
165
166	acquire_lock
167
168	if ! command -v docker &>/dev/null; then
169	    error "Docker command not found"
170	    exit 1
171	fi
172
173	platform_require_cmd \
174	    "/usr/local/emhttp/plugins/dynamix/scripts/notify" \
175	    "" "" \
176	    "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
177
178	detect_hosts
179
180	# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller
181	if ! check_rsync_enabled; then
182	    warn "RSYNC_ENABLED=false — exiting cleanly"
183	    exit 0
184	fi
185
186	# Blocklist gate — refuse to sync with a partner blocked after offboard
187	BLOCKLIST_FILE="${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR:-/boot/config}/partnership_blocklist.db}"
188	if [[ -f "$BLOCKLIST_FILE" ]] && grep -q "^${REMOTE_SERVER_NAME}|" "$BLOCKLIST_FILE" 2>/dev/null; then
189	    error "Rsync blocked — $REMOTE_SERVER_NAME is on the partnership blocklist"
190	    error "Re-onboard the partnership to restore access: partnership_manager.sh --onboard"
191	    exit 1
192	fi
193
194	resolve_remote_ip
195
196	# ── Profile inference ─────────────────────────────────────────────────────────────────────────
197	if [[ -n "$PROFILE_OVERRIDE" ]]; then
198	    PROFILE_NAME="$PROFILE_OVERRIDE"
199	    log "Profile override: $PROFILE_NAME"
200	else
201	    PROFILE_NAME=$(basename "$DIRECTORY" | tr '[:upper:]' '[:lower:]')
202	    log "Profile inferred: $PROFILE_NAME"
203	fi
204
205	# Acquire per-profile lock and check global concurrent limit
206	acquire_rsync_lock "$PROFILE_NAME"
207
208	# Tee all output to a live log file for the Varaverk UI
209	VV_LIVE_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.log"
210	VV_LAST_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.last.log"
211	: > "$VV_LIVE_LOG"
212	exec 1> >(tee -a "$VV_LIVE_LOG") 2>&1
213
214	# ── Load profile settings ─────────────────────────────────────────────────────────────────────
215	BW_LIMIT=${PROFILE_BW_LIMIT[$PROFILE_NAME]:-$BW_LIMIT}
216	RETRY_COUNT=${PROFILE_RETRY_COUNT[$PROFILE_NAME]:-$RETRY_COUNT}
217	SLEEP=${PROFILE_SLEEP[$PROFILE_NAME]:-$SLEEP}

# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
    error "Must be run as root"
    exit 1
fi

acquire_lock

if ! command -v docker &>/dev/null; then
    error "Docker command not found"
    exit 1
fi

platform_require_cmd \
    "/usr/local/emhttp/plugins/dynamix/scripts/notify" \
    "" "" \
    "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"

detect_hosts

# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller
if ! check_rsync_enabled; then
    warn "RSYNC_ENABLED=false — exiting cleanly"
    exit 0
fi

# Blocklist gate — refuse to sync with a partner blocked after offboard
BLOCKLIST_FILE="${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR:-/boot/config}/partnership_blocklist.db}"
if [[ -f "$BLOCKLIST_FILE" ]] && grep -q "^${REMOTE_SERVER_NAME}|" "$BLOCKLIST_FILE" 2>/dev/null; then
    error "Rsync blocked — $REMOTE_SERVER_NAME is on the partnership blocklist"
    error "Re-onboard the partnership to restore access: partnership_manager.sh --onboard"
    exit 1
fi

resolve_remote_ip

# ── Profile inference ─────────────────────────────────────────────────────────────────────────
if [[ -n "$PROFILE_OVERRIDE" ]]; then
    PROFILE_NAME="$PROFILE_OVERRIDE"
    log "Profile override: $PROFILE_NAME"
else
    PROFILE_NAME=$(basename "$DIRECTORY" | tr '[:upper:]' '[:lower:]')
    log "Profile inferred: $PROFILE_NAME"
fi

# Acquire per-profile lock and check global concurrent limit
acquire_rsync_lock "$PROFILE_NAME"

# Tee all output to a live log file for the Varaverk UI
VV_LIVE_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.log"
VV_LAST_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.last.log"
: > "$VV_LIVE_LOG"
exec 1> >(tee -a "$VV_LIVE_LOG") 2>&1

# ── Load profile settings ─────────────────────────────────────────────────────────────────────
BW_LIMIT=${PROFILE_BW_LIMIT[$PROFILE_NAME]:-$BW_LIMIT}
RETRY_COUNT=${PROFILE_RETRY_COUNT[$PROFILE_NAME]:-$RETRY_COUNT}
SLEEP=${PROFILE_SLEEP[$PROFILE_NAME]:-$SLEEP}


Found the bug. Line 212 sets up `exec 1> >(tee) 2>&1` — both stdout and stderr point to the tee pipe. Line 456 does `exec 1>&-; wait` — closes stdout but **leaves stderr still holding the pipe open**. Tee never gets EOF, `wait` blocks forever, and the lock is never released.

Let me check the trap setup and then fix this.


grep -n "_register_lock\|_lock_cleanup\|trap.*EXIT\|trap.*cleanup" /boot/config/plugins/varaverk/common.sh | head -20

Find trap/cleanup registration in common.sh


1360:_register_lock() {
1362:    trap '_release_all_locks' EXIT
1456:    _register_lock "$lockfile"
1515:    _register_lock "$profile_lock"


1355	    local lf
1356	    for lf in "${_LOCK_FILES[@]}"; do
1357	        [[ -f "$lf" ]] && rm -f "$lf"
1358	    done
1359	}
1360	_register_lock() {
1361	    _LOCK_FILES+=("$1")
1362	    trap '_release_all_locks' EXIT
1363	}
1364
1365	# Internal — script name used as lock identifier
1366	_lock_name() {
1367	    basename "${BASH_SOURCE[1]:-$0}" .sh
1368	}
1369
1370	# Internal — lock file path for this script
1371	_lock_file() {
1372	    echo "$LOCK_DIR/${1:-$(_lock_name)}.lock"
1373	}
1374
1375	# Internal — release lock on exit (registered via EXIT trap)
1376	_release_on_exit() {
1377	    local lockfile="$1"
1378	    [[ -f "$lockfile" ]] && rm -f "$lockfile"
1379	}
1380
1381	# Acquire exclusive lock for this script.
1382	# Usage: acquire_lock [strict|wait|continuous]
1383	acquire_lock() {
1384	    local mode="${1:-strict}"

    local lf
    for lf in "${_LOCK_FILES[@]}"; do
        [[ -f "$lf" ]] && rm -f "$lf"
    done
}
_register_lock() {
    _LOCK_FILES+=("$1")
    trap '_release_all_locks' EXIT
}

# Internal — script name used as lock identifier
_lock_name() {
    basename "${BASH_SOURCE[1]:-$0}" .sh
}

# Internal — lock file path for this script
_lock_file() {
    echo "$LOCK_DIR/${1:-$(_lock_name)}.lock"
}

# Internal — release lock on exit (registered via EXIT trap)
_release_on_exit() {
    local lockfile="$1"
    [[ -f "$lockfile" ]] && rm -f "$lockfile"
}

# Acquire exclusive lock for this script.
# Usage: acquire_lock [strict|wait|continuous]
acquire_lock() {
    local mode="${1:-strict}"


grep -n "_release_rsync_on_exit\|release_rsync" /boot/config/plugins/varaverk/common.sh | head -10

Find rsync lock release registration


1520:_release_rsync_on_exit() {


1510	    fi
1511
1512	    # Acquire profile lock and increment counter
1513	    echo "$$:rsync_${profile}" > "$profile_lock"
1514	    echo $(( current_count + 1 )) > "$RSYNC_COUNT_FILE"
1515	    _register_lock "$profile_lock"
1516	    log "$ICON_LOCK rsync lock acquired: profile '$profile' (PID $$, active: $(( current_count + 1 ))/$RSYNC_MAX_CONCURRENT)"
1517	}
1518
1519	# Internal — release rsync lock and decrement global counter on exit
1520	_release_rsync_on_exit() {
1521	    local profile_lock="$1"
1522	    [[ -f "$profile_lock" ]] && rm -f "$profile_lock"
1523	    if [[ -f "$RSYNC_COUNT_FILE" ]]; then
1524	        local count
1525	        count=$(cat "$RSYNC_COUNT_FILE" 2>/dev/null || echo 1)
1526	        count=$(( count - 1 ))
1527	        [[ "$count" -lt 0 ]] && count=0
1528	        echo "$count" > "$RSYNC_COUNT_FILE"
1529	    fi
1530	}
1531
1532	# ==============================================================================================
1533	# ── PATH TRANSLATION ──────────────────────────────────────────────────────────────────────────
1534	# ==============================================================================================

    fi

    # Acquire profile lock and increment counter
    echo "$$:rsync_${profile}" > "$profile_lock"
    echo $(( current_count + 1 )) > "$RSYNC_COUNT_FILE"
    _register_lock "$profile_lock"
    log "$ICON_LOCK rsync lock acquired: profile '$profile' (PID $$, active: $(( current_count + 1 ))/$RSYNC_MAX_CONCURRENT)"
}

# Internal — release rsync lock and decrement global counter on exit
_release_rsync_on_exit() {
    local profile_lock="$1"
    [[ -f "$profile_lock" ]] && rm -f "$profile_lock"
    if [[ -f "$RSYNC_COUNT_FILE" ]]; then
        local count
        count=$(cat "$RSYNC_COUNT_FILE" 2>/dev/null || echo 1)
        count=$(( count - 1 ))
        [[ "$count" -lt 0 ]] && count=0
        echo "$count" > "$RSYNC_COUNT_FILE"
    fi
}

# ==============================================================================================
# ── PATH TRANSLATION ──────────────────────────────────────────────────────────────────────────
# ==============================================================================================


Clear picture now. Here's what happened and what needs to happen:

**Root cause**: Line 212 of `rsync.sh` does `exec 1> >(tee) 2>&1` — both stdout **and** stderr point to the tee pipe. Line 456 does `exec 1>&-; wait` which closes stdout but leaves stderr still holding the pipe open. Tee never gets EOF, `wait` blocks forever, the process never exits, and the locks stay held. The rsync itself finished in 9 seconds last night around 4:30pm — this has been a zombie wrapper ever since.

**Three things to do:**

1. Kill the stuck process (SIGTERM triggers the EXIT trap which cleans up the lock files)
2. Fix the one-liner in `rsync.sh`
3. Manually kick the two failed syncs

The fix is trivial:


454
455	# Flush tee and preserve log for UI
456	exec 1>&-; wait
457	cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
458	rm -f "$VV_LIVE_LOG"

# Flush tee and preserve log for UI
exec 1>&-; wait
cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
rm -f "$VV_LIVE_LOG"


Edit

# Flush tee and preserve log for UI
exec 1>&-; wait

# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF
exec 1>&- 2>&-; wait


The file /boot/config/plugins/varaverk/Rsync/rsync.sh has been updated successfully. (file state is current in your context — no need to Read it back)

# Flush tee and preserve log for UI
exec 1>&-; wait

# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF
exec 1>&- 2>&-; wait

echo "━━━━━━━━━━━━━━━━━━━━━━━"

+# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF

cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null


Now kill the stuck process and verify the locks clear:


Bash

kill -TERM 2486476 && sleep 1 && ps -p 2486476 2>/dev/null && echo "still alive" || echo "process exited"

SIGTERM the stuck rsync wrapper and confirm it exits


    PID TTY          TIME CMD
process exited

    PID TTY          TIME CMD
process exited


ls -la /tmp/unraid_locks/rsync*.lock /tmp/unraid_locks/rsync_active_count 2>/dev/null || echo "all rsync locks cleared"

Confirm lock files were cleaned up by the EXIT trap


-rw-r--r-- 1 root root 14 Jun  7 09:24 /tmp/unraid_locks/rsync.lock
-rw-r--r-- 1 root root  2 Jun  6 16:30 /tmp/unraid_locks/rsync_active_count


The profile lock cleared but `rsync.lock` (the global script lock) and `rsync_active_count` are still around. Let me check.


cat /tmp/unraid_locks/rsync.lock; echo; cat /tmp/unraid_locks/rsync_active_count

Check remaining lock file contents


3005590:rsync

1


ps -p 3005590 -o pid,etime,cmd 2>/dev/null || echo "PID 3005590 not found — stale lock"

Check if rsync.lock PID is alive


    PID     ELAPSED CMD
3005590       00:21 bash /boot/config/plugins/varaverk/Orchestrators/../Rsync/rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback


That `rsync.lock` is actually from a freshly-started Emby fallback sync (PID 3005590, 21 seconds old) — the system started syncing on its own. Locks are clean.

Let me check that the fix is in order and do a quick sanity check on the count file:


cat /tmp/unraid_locks/rsync_active_count; ls /tmp/unraid_locks/rsync_*.lock 2>/dev/null

Confirm active count matches live lock files


1
/tmp/unraid_locks/rsync_emby-fallback.lock


Everything is consistent: count=1, one active profile lock. The weekly syncs (Emby and Critical-Data) will need to be run manually since they were blocked this morning — do you want me to kick them now, or wait for the in-progress emby-fallback sync to finish first?

**Summary of the bug and fix:**

- `rsync.sh` redirects both stdout and stderr into a tee pipe at startup (`exec 1> >(tee) 2>&1`)
- The shutdown sequence only closed stdout (`exec 1>&-`), leaving stderr still holding the pipe open
- Tee blocked waiting for EOF, `wait` blocked waiting for tee, the bash wrapper never exited, and the locks were never released
- Fix: `exec 1>&- 2>&-; wait` — closes both ends so tee gets a proper EOF


they will run next week. i have 2 users on


Makes sense — the Emby sync stops containers, bad timing with active users. Want me to commit the fix so it's in place for next week's run?


yeah commit it


cd /boot/config/plugins/varaverk && git diff Rsync/rsync.sh

Show the diff to commit


diff --git a/Rsync/rsync.sh b/Rsync/rsync.sh
index cfe2411..f7468cf 100755
--- a/Rsync/rsync.sh
+++ b/Rsync/rsync.sh
@@ -452,8 +452,8 @@ else
 fi
 echo "━━━━━━━━━━━━━━━━━━━━━━━"

-# Flush tee and preserve log for UI
-exec 1>&-; wait
+# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF
+exec 1>&- 2>&-; wait
 cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
 rm -f "$VV_LIVE_LOG"
Shell cwd was reset to /root


cd /boot/config/plugins/varaverk && git add Rsync/rsync.sh && git commit -m "Fix rsync wrapper hanging forever after sync completes

exec 1>&- closed stdout but left stderr holding the tee pipe open,
so tee never got EOF and wait blocked indefinitely — keeping the
process alive and the locks held until manually killed."

Commit the rsync fix


[main 80b1ee4] Fix rsync wrapper hanging forever after sync completes
 1 file changed, 2 insertions(+), 2 deletions(-)
Shell cwd was reset to /root


Committed. Next Sunday the Emby and Critical-Data syncs will run clean.


⚠  [WARN]  ❌ MISSING: theme.mp3
⚠  [WARN]  ❌ MISSING: Pil's Adventures (2021)-trailer.mkv
⚠  [WARN]  ❌ MISSING: A Wrinkle in Time (2018) - [Remux-1080p].mkv
⚠  [WARN]  ❌ MISSING: fanart.jpg
⚠  [WARN]  Kids_Movies — match: 3  missing: 7  mismatch: 0

━━━ ✔ Kids_Tv_Shows ━━━
⚠  [WARN]  ❌ MISSING: As Told By Ginger (2000) - S02E15 - Mommie Nearest HDTV-1080p.mkv
⚠  [WARN]  ❌ MISSING: Chip 'n Dale Rescue Rangers (1989) - S02E07 - The Luck Stops Here Bluray-1080p.mkv
⚠  [WARN]  Kids_Tv_Shows — match: 8  missing: 2  mismatch: 0

━━━ ✔ Movies ━━━
⚠  [WARN]  ❌ MISSING: theme.mp3
⚠  [WARN]  ❌ MISSING: Split (2017) - [Bluray-1080p].mkv
⚠  [WARN]  ❌ MISSING: clearart.png
⚠  [WARN]  ❌ MISSING: Meet the Parents (2000) 2-trailer.mkv
⚠  [WARN]  ❌ MISSING: The Boy (2016)-trailer.mkv
⚠  [WARN]  ❌ MISSING: Get Him to the Greek (2010) - Unrated [Remux-1080p].mkv
⚠  [WARN]  ❌ MISSING: The Old Guard 2 (2025) 2-trailer.mkv
⚠  [WARN]  ❌ MISSING: Day of the Dead Bloodline (2017) 2-trailer.mkv
⚠  [WARN]  Movies — match: 2  missing: 8  mismatch: 0

━━━ ✔ Music ━━━
⚠  [WARN]  Music not found locally — skipping

━━━ ✔ Music_Videos ━━━

━━━ ✔ Nextcloud ━━━
❌ [ERROR] MISMATCH: apps.json
❌ [ERROR]   local:  1e8e710ce97313f9a0179f698d19e7e7
❌ [ERROR]   remote: 09ea68e23ea827c9a4da7a09d96366c3
⚠  [WARN]  Nextcloud — match: 9  missing: 0  mismatch: 1

━━━ ✔ stand-up_comedy ━━━
⚠  [WARN]  ❌ MISSING: Matthew Steven Rife (2023) - [Bluray-720p].mp4
⚠  [WARN]  ❌ MISSING: The Roast of Tom Brady (2024) - [WEBDL-1080p].mkv
⚠  [WARN]  ❌ MISSING: Anthony Jeselnik Thoughts and Prayers (2015) - [WEBRip-1080p].mp4
⚠  [WARN]  ❌ MISSING: Joe Rogan Rocky Mountain High (2014) - [WEBRip-1080p Proper].mkv
⚠  [WARN]  ❌ MISSING: Comedy Central Roast of Pamela Anderson (2005) - [DVD].avi
⚠  [WARN]  ❌ MISSING: Anthony Jeselnik Bones and All (2024) - [WEBRip-1080p].mkv
⚠  [WARN]  ❌ MISSING: Tom Segura Sledgehammer (2023) - [WEBRip-1080p].mkv
⚠  [WARN]  ❌ MISSING: Anthony Jeselnik Fire in the Maternity Ward (2019) - [WEBDL-1080p].mkv
⚠  [WARN]  ❌ MISSING: Bob Saget That Ain't Right (2007) - [WEBDL-1080p].mkv
⚠  [WARN]  ❌ MISSING: Comedy Central Roast of Charlie Sheen (2011) - [WEBDL-1080p].mkv
⚠  [WARN]  stand-up_comedy — match: 0  missing: 10  mismatch: 0

━━━ ✔ Sports ━━━
⚠  [WARN]  ❌ MISSING: .NFL 2020-Super-Bowl-LIV_SF@KC.mkv.Uc2bAj
⚠  [WARN]  ❌ MISSING: NFL 2014 Super Bowl XLVIII _ Seahawks vs Broncos (720p_60fps_H264_AC3).mkv
⚠  [WARN]  ❌ MISSING: NFL 2007 Super Bowl XLI _ Colts vs. Bears (720p_30fps_H264-192kbit_AAC).mkv
⚠  [WARN]  ❌ MISSING: NFL 2010 Super Bowl XLIV _ Saints.vs.Colts  (720p_60fps_H264_AC3).mkv
⚠  [WARN]  ❌ MISSING: NFL 2018_SuperBowl_LII_PHI@NE_720p60.mkv
⚠  [WARN]  ❌ MISSING: NFL 2005 - Super Bowl XXXIX - 06.02.2005 - New England Patriots vs. Philadelphia Eagles.mkv
⚠  [WARN]  ❌ MISSING: NFL 2012 Super Bowl XLVI _ Giants vs Patriots (720p_30fps_H264_AC3).mkv
⚠  [WARN]  ❌ MISSING: NFL 2019.Super.Bowl.LIII.03.02.2019.Patriots.vs.Rams.720p.60fps.mkv
⚠  [WARN]  ❌ MISSING: NFL 2008 Super Bowl XLII New York Giants vs New England Patriots Full Game 720p 30fps En.mkv
⚠  [WARN]  ❌ MISSING: NFL 2004 Super Bowl XXXVIII - 01.02.2004 - New England Patriots vs. Carolina Panthers.mkv
⚠  [WARN]  Sports — match: 0  missing: 10  mismatch: 0

━━━ ✔ Anime_Shows-Old ━━━
⚠  [WARN]  ❌ MISSING: Gintama (2006) - S07E48 - 313 - Undelivered Mail Bluray-1080p.mkv
⚠  [WARN]  ❌ MISSING: The Rising of the Shield Hero (2019) - S03E08 - Dragons Den Bluray-1080p.mkv
⚠  [WARN]  ❌ MISSING: My Hero Academia (2016) - S07E12 - Those Who Defend Those Who Violate Bluray-1080p.mkv
⚠  [WARN]  ❌ MISSING: Bungo Stray Dogs (2016) - S02E07 - Fitzgerald Rising Bluray-1080p.mkv
⚠  [WARN]  Anime_Shows-Old — match: 6  missing: 4  mismatch: 0

━━━ ✔ Anime_Movies-Old ━━━

━━━ ✔ Anime_Movies ━━━

━━━ ✔ Anime_Shows ━━━

━━━━━ 📋 BACKUP VERIFY SUMMARY ━━━━━
🖥    My ID:     HOST1 (unRAID-Gmer4Lfe)
🖥    Remote:    HOST2 (unRAID-Jayred365)
✔  Checked:   86 files
⏱    Duration:  18m32s

✅ Match:     44
⚠  [WARN]  Missing:   41
❌   Mismatch:  1
❌ Status: ISSUES FOUND — 7 share(s) need attention: Kids_Movies Kids_Tv_Shows Movies Nextcloud stand-up_comedy Sports Anime_Shows-Old
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
❌ [ERROR] backup_verify.sh — failed (exit 1)

━━━ 📶 Bandwidth Report — 2026-06-07 07:19:01 ━━━
🖥 HOST1 (unRAID-Gmer4Lfe)

ℹ  [INFO]  Log covers: 2026-04-15 → 2026-06-06 (640 runs)

━━━ 📶 Per-Profile Summary ━━━
  anime_movies-old         1 runs  avg 0m3s  failed: 0
  anime_shows              1 runs  avg 0m13s  failed: 0
  anime_shows-old          1 runs  avg 2m32s  failed: 0
  arrs_stack              21 runs  avg 5m5s  failed: 0
  books                    2 runs  avg 0m0s  failed: 0
  critical-data           75 runs  avg 0m9s  failed: 0
  critical-fallback      123 runs  avg 0m11s  failed: 0
  emby                   145 runs  avg 33m22s  failed: 0
  emby-failover          105 runs  avg 0m8s  failed: 0
  emby-fallback          122 runs  avg 0m11s  failed: 0
  important-data          40 runs  avg 0m54s  failed: 0
  kids_movies              2 runs  avg 2m45s  failed: 0
  kids_tv_shows            2 runs  avg 2868m17s  failed: 0

━━━ 📶 Last 7 Days ━━━
  ⏱  2026-06-01 (Mon) — no syncs
  🏁  2026-06-02 (Tue) — 8 runs / 4m8s total
  🏁  2026-06-03 (Wed) — 84 runs / 13m57s total
  🏁  2026-06-04 (Thu) — 22 runs / 3m57s total
  🏁  2026-06-05 (Fri) — 64 runs / 11m42s total
  🏁  2026-06-06 (Sat) — 67 runs / 12m42s total
  ⏱  2026-06-07 (Sun) — no syncs

━━━━━ 📋 BANDWIDTH SUMMARY ━━━━━
🖥       Identity:  HOST1 (unRAID-Gmer4Lfe)
📶  Runs:      640 (640 success / 0 failed)
⏱       Total:     7d 12h 0m
⏱       Period:    2026-04-15 → 2026-06-06
⚙       Retention: 90 days
⏱       Generated: 0s
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

━━━ ⚙ Setup ━━━

━━━ 🎬 Emby Session Report — 2026-06-07 07:19:01 ━━━
🖥  HOST1 (unRAID-Gmer4Lfe)
⏱  Period: Last 7 days

━━━ 🎬 Active Sessions ━━━
  🎬 Active streams:  3
  🎬 Direct play:     2
  🎬 Transcoding:     1

  Now playing:
    Sunny → Sundance TV [transcode]
    SammySosa412 → Something, Something, Something, Dark Side [direct]
    Sunny → Cinemax 5Starmax [direct]

━━━ 🎬 Library ━━━
  🎬 Movies:    2986
  🎬 Episodes:  94620
  🎬 Songs:     134058

━━━ 🎬 Activity — Last 7 Days ━━━
  🎬 Total play events:  0

━━━ 🎬 Top 10 Content ━━━
⚠  [WARN]  Could not fetch top content
  No recent play history found

━━━ 🎬 Most Active Users ━━━
  🎬 Total users: 24
  Sunny — last active: 2026-06-07
  SammySosa412 — last active: 2026-06-07
  choak178 — last active: 2026-06-07
  Aleashadukes5901 — last active: 2026-06-07
  WRX15235 — last active: 2026-06-07

━━━ 💨 Transcode Status ━━━
  💨  Ramdisk usage:  1.73GB / 10G
  🔗 Symlink target: /mnt/ramdisk_transcodes

━━━━━ 📋 EMBY REPORT SUMMARY ━━━━━
🖥    Identity:  HOST1 (unRAID-Gmer4Lfe)
🎬    Server:    Emby-Gmer4Lfe (v4.9.5.0)
🎬    Active:    3 streams (2 direct / 1 transcode)
🎬    Library:   2986 movies  94620 episodes  134058 songs
🎬    Period:    0 play events in last 7 days
⏱    Duration:  18s
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

━━━ 📰 Health Digest — 2026-06-07 07:19:32 ━━━
🖥 HOST1 (unRAID-Gmer4Lfe)

  🔀 Fallback: NORMAL
  🟢 Skip list: empty ✅
  🐾 Container strikes: 1 active — daemon_confirmed_down:false
  🚨 System strikes: 4 active — failed_containers:1 kernel_oops_count=00 mdstat_errors=0 watchdog_cycle=1780830954
  💨 Transcodes: 1.73GB used → /mnt/ramdisk_transcodes
  💨 Transcode week: peak 6.65GB | flips: 0 | sessions: 4543 ramdisk / 0 SSD | cleaned: 0 files
  📶 Yesterday: 0.00GB
  🔒 Certificates: all healthy ✅

━━━━━ 📋 DIGEST SUMMARY ━━━━━
🖥    Identity: HOST1 (unRAID-Gmer4Lfe)
📰  Profile:  weekly
❌   Issues:   0
⚠    Findings: 2
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Digest sent

━━━━━ ☕ COFFEE REPORT SUMMARY ━━━━━
🖥  Host:    HOST1
⏱  Done:    2026-06-07 07:19:32
✅  Passed:  zfs_memory_snapshot.sh smart_health.sh bandwidth_monitor.sh emby_session_report.sh weekly_health_digest.sh
❌  Failed:  cert_monitor.sh backup_verify.sh


⚠  [WARN]  ❌ MISSING: theme.mp3 ⚠  [WARN]  ❌ MISSING: Pil's Adventures (2021)-trailer.mkv ⚠  [WARN]  ❌ MISSING: A Wrinkle in Time (2018) - [Remux-1080p].mkv ⚠  [WARN]  ❌ MISSING: fanart.jpg ⚠  [W…


Two things worth flagging from this report before I dig in:

1. `cert_monitor.sh` failed — that's the one that matters urgently
2. The `kids_tv_shows` average of **2868 minutes** in the bandwidth report is almost certainly the same tee deadlock bug we just fixed hitting a different profile run

Let me check the cert failure first.


find /var/log/varaverk -name "cert*" 2>/dev/null; find /var/log -name "cert*" 2>/dev/null | head -10

Find cert monitor log


Read cert status state file


(Bash completed with no output)


`Gmer4Lfe.com` cert is fine — 55 days out (expires Aug 1). The cert_monitor failure is `Gmer4Lfe.us` returning null because HOST2 is still being rebuilt. It's a false alarm, but the script exits 1 if any domain fails.

The rest of the report is pretty clean given the context:

**Expected / explained by DAILY_RSYNC_ENABLED=false:**
- The 41 missing files across Sports, stand-up_comedy, Kids_Movies, etc. are files added to HOST1 since daily sync was paused. They'll land on HOST2 when you re-enable it.
- The Nextcloud `apps.json` mismatch is the same story — diverged while sync was off.

**Explained by the bug we just fixed:**
- `kids_tv_shows: avg 2868m17s` in the bandwidth report — 47+ hours per run. That's the same tee deadlock on a different profile. Now fixed.

**Actually worth a look:**
- `🐾 Container strikes: 1 active` — which container?
- `🚨 System strikes: 4 active, failed_containers:1` — there's a container that's failed

Want me to pull up the watchdog state on those strikes?


yup


sqlite3 /boot/config/plugins/varaverk/State_Files/container_watchdog_state.db "SELECT name, type, value, updated_at FROM watchdog_state ORDER BY updated_at DESC LIMIT 30;" 2>/dev/null

Query container watchdog state DB


sqlite3 /boot/config/plugins/varaverk/State_Files/system_watchdog_state.db "SELECT name, type, value, updated_at FROM watchdog_state ORDER BY updated_at DESC LIMIT 30;" 2>/dev/null

Query system watchdog state DB


Exit code 1


sqlite3 /boot/config/plugins/varaverk/State_Files/container_watchdog_state.db ".tables" 2>/dev/null; sqlite3 /boot/config/plugins/varaverk/State_Files/system_watchdog_state.db ".tables" 2>/dev/null

Check actual table names in watchdog DBs


Check file types of watchdog state files


/boot/config/plugins/varaverk/State_Files/container_watchdog_state.db: ASCII text
/boot/config/plugins/varaverk/State_Files/system_watchdog_state.db:    ASCII text
/boot/config/plugins/varaverk/State_Files/docker_watchdog_failed.db:   empty


cat /boot/config/plugins/varaverk/State_Files/container_watchdog_state.db; echo "---"; cat /boot/config/plugins/varaverk/State_Files/system_watchdog_state.db

Read watchdog state files


daemon_confirmed_down:false
NginxProxyManager:0
Lldap-Gmer4Lfe:0
Mariadb-Authelia:0
Mariadb-Authelia-Secondary:0
Redis-Authelia:0
Redis-Authelia-Secondary:0
Authelia:0
Authelia-Secondary:0
Code-Server_cpu:0
Emby_cpu:0
Emby_http:0
Jellyfin_docker:0
Emby_docker:0
EmyCredits-Ocr_docker:0
slskd_docker:0
SABnzbd-Gmer4Lfe_docker:0
SeerrFin_docker:0
Soularr_docker:0
NextCloud_docker:0
Readarr_docker:0
Profilarr_docker:0
Pinchflat_docker:0
Organizrv2-Gmer4Lfe_docker:0
Dispatcharr-Iptv-Users_docker:0
Dispatcharr-Basic_docker:0
Dispatcharr_docker:0
Code-Server_docker:0
Sonarr_docker:0
Trailarr_docker:0
Radarr_docker:0
Prowlarr_docker:0
Lidarr_docker:0
Cleanuparr_docker:0
Bazarr_docker:0
Authelia-Secondary_docker:0
Authelia_docker:0
Redis-Authelia-Secondary_docker:0
Redis-Authelia_docker:0
Mariadb-Authelia-Secondary_docker:0
Mariadb-Authelia_docker:0
NginxProxyManager_docker:0
Lldap-Gmer4Lfe_docker:0
degoog_docker:0
Qbittorrent-Gmer4Lfe_docker:0
AdGuard-Home_docker:0
Collabora-CODE_docker:0
Unraid-progressive-web-app_docker:0
Immich-Gmer4Lfe_docker:0
Koffan_docker:0
ChannelTube_docker:0
Gitea_docker:0
UptimeKuma-Gmer4Lfe_docker:0
honeygain_docker:0
DHCP-Netboot_docker:0
speedtest-tracker_docker:0
VaultWarden-Gmer4Lfe_docker:0
LocalAI_docker:0
Krusader_docker:0
Jellyseerr_docker:0
ErsatzTV-Emby_docker:0
Postgres-NextCloud_docker:0
PostgreSQL_Immich_docker:0
7DaysToDie_docker:0
Healarr_docker:0
Playerr_docker:0
Redis-Immich_docker:0
Fetcharr_docker:0
Gmer4Lfe.com_docker:0
Emby_api:0
Gmer4Lfe.us_docker:0
claudeclaw_docker:0
musicseerr_docker:0
Seerr_docker:0
---
failed_containers:1
kernel_oops_count=00
ram:0
mdstat_errors=0
sshd:0
watchdog_cycle=1780839027


Container strikes are all at 0 now (auto-resolved since the digest ran). The `failed_containers:1` in the system state is the one still standing. Let me find which container it flagged.


sqlite3 /boot/config/plugins/varaverk/State_Files/container_restart_history.db ".tables" 2>/dev/null && sqlite3 /boot/config/plugins/varaverk/State_Files/container_restart_history.db "SELECT * FROM restarts ORDER BY timestamp DESC LIMIT 10;" 2>/dev/null

Check recent container restart history


grep -i "failed\|ERROR\|strike\|restart\|down" /boot/config/plugins/varaverk/State_Files/watchdog_orchestrator.log 2>/dev/null | tail -30

Check watchdog orchestrator log for failures


🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings


`emby-test` exited with code 127 (command not found — likely a bad entrypoint or image issue) and every watchdog restart attempt is failing. It's been looping on this for hours.


docker inspect emby-test --format '{{.State.Status}} | exit={{.State.ExitCode}} | started={{.State.StartedAt}} | finished={{.State.FinishedAt}}' 2>/dev/null

Inspect emby-test container state


Get last 20 lines of emby-test logs


Error: Exit code 1
Error response from daemon: No such container: emby-test


Container was removed but the watchdog still has it in its list. Let me find where `emby-test` is defined.


grep -rn "emby-test" /boot/config/plugins/varaverk/ 2>/dev/null

Find all references to emby-test in the varaverk config


/boot/config/plugins/varaverk/Configurations/host1.conf:333:        "emby-test"                 # broken test container (exit 127 — bad image)
/boot/config/plugins/varaverk/Configurations/host1.conf.bak:333:        "emby-test"                 # broken test container (exit 127 — bad image)


325	# Watchdog leaves these alone entirely — no restart attempts, no crash loop tracking.
326	    HOST1_WATCHDOG_SCAN_IGNORE=(
327	        "DashGate"
328	        "PIA-WG-Config-Generator"
329	        "Aperture"
330	        "Aperture-Kids"
331	        "pgvector-18-Apeture-Kids"
332	        "Pgvector18-Aperture"
333	        "emby-test"                 # broken test container (exit 127 — bad image)
334	    )
335
336	# Dependency ordering — skip restarting a container if its dependency is also down.
337	# Prevents watchdog from restarting Authelia before Mariadb is back up.
338	# SPACE-SEPARATED STRINGS — converted to array at runtime.
339	    declare -A HOST1_WATCHDOG_DEPENDENCIES=(
340	        ["Authelia"]="Mariadb-Authelia Redis-Authelia"
341	        ["Authelia-Secondary"]="Mariadb-Authelia Redis-Authelia-Secondary"
342	        ["NextCloud"]="Postgres-NextCloud"
343	    )
344

# Watchdog leaves these alone entirely — no restart attempts, no crash loop tracking.
    HOST1_WATCHDOG_SCAN_IGNORE=(
        "DashGate"
        "PIA-WG-Config-Generator"
        "Aperture"
        "Aperture-Kids"
        "pgvector-18-Apeture-Kids"
        "Pgvector18-Aperture"
        "emby-test"                 # broken test container (exit 127 — bad image)
    )

# Dependency ordering — skip restarting a container if its dependency is also down.
# Prevents watchdog from restarting Authelia before Mariadb is back up.
# SPACE-SEPARATED STRINGS — converted to array at runtime.
    declare -A HOST1_WATCHDOG_DEPENDENCIES=(
        ["Authelia"]="Mariadb-Authelia Redis-Authelia"
        ["Authelia-Secondary"]="Mariadb-Authelia Redis-Authelia-Secondary"
        ["NextCloud"]="Postgres-NextCloud"
    )


It's already in `HOST1_WATCHDOG_SCAN_IGNORE` — but the watchdog is still trying to restart it. Something's wrong with how the ignore list is being applied.


grep -n "SCAN_IGNORE\|scan_ignore\|ignore" /boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh | head -20

Find SCAN_IGNORE usage in docker_watchdog


32:#   Containers in WATCHDOG_SCAN_IGNORE are excluded.
161:#   HOST*_WATCHDOG_SCAN_IGNORE
163:#       WATCHDOG_SCAN_IGNORE
280:    echo "$ICON_WATCHDOG   Ignore:         ${WATCHDOG_SCAN_IGNORE[*]:-none}"
619:    # Rebuild ignore map each cycle (config may have changed)
621:    for c in "${WATCHDOG_SCAN_IGNORE[@]:-}"; do


615	    # OOM handled set — tracks containers already handled for OOM this cycle
616	    # Prevents restart loop from OOMKilled flag persisting after restart
617	    declare -A OOM_HANDLED
618
619	    # Rebuild ignore map each cycle (config may have changed)
620	    declare -A IGNORE_MAP
621	    for c in "${WATCHDOG_SCAN_IGNORE[@]:-}"; do
622	        [[ -n "$c" ]] && IGNORE_MAP["$c"]=1
623	    done
624
625	    # ── Skip list visibility ─────────────────────────────────────────────────────────────────
626	    _skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
627	    [[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
628
629	    # ── Docker daemon health check — first check every run ──────────────────────────────────
630	    # If daemon is hung all container operations will fail — check first, skip run if down
631	    if ! check_docker_daemon; then
632	        exit 0
633	    fi
634
635	    # ── Parity check — skip restarts during parity ───────────────────────────────────────────
636	    if is_parity_running; then
637	        echo "Parity in progress — restart actions skipped ($(date '+%H:%M:%S'))"
638	        exit 0
639	    fi
640
641	    # ── RAM emergency check — resource_watchdog.sh managing containers ────────────────────────
642	    # If resource_watchdog.sh has triggered a hard RAM shutdown, defer all container
643	    # management this run to prevent undoing the emergency stop and re-pressuring RAM.
644	    if ! check_resource_watchdog_state; then
645	        MEM_KB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
646	        MEM_GB=$(( MEM_KB / 1024 / 1024 ))
647	        warn "RAM emergency active (${MEM_GB}GB free) — resource_watchdog.sh managing containers"
648	        warn "Deferring all container restart logic this run"
649	        log "Waiting for RAM to recover above ${RW_RAM_RECOVER_GB:-20}GB before resuming"
650	        exit 0
651	    fi
652
653	    # ── Startup grace period ──────────────────────────────────────────────────────────────────
654	    UPTIME_SECONDS=$(awk '{print int($1)}' /proc/uptime)

    # OOM handled set — tracks containers already handled for OOM this cycle
    # Prevents restart loop from OOMKilled flag persisting after restart
    declare -A OOM_HANDLED

    # Rebuild ignore map each cycle (config may have changed)
    declare -A IGNORE_MAP
    for c in "${WATCHDOG_SCAN_IGNORE[@]:-}"; do
        [[ -n "$c" ]] && IGNORE_MAP["$c"]=1
    done

    # ── Skip list visibility ─────────────────────────────────────────────────────────────────
    _skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
    [[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"

    # ── Docker daemon health check — first check every run ──────────────────────────────────
    # If daemon is hung all container operations will fail — check first, skip run if down
    if ! check_docker_daemon; then
        exit 0
    fi

    # ── Parity check — skip restarts during parity ───────────────────────────────────────────
    if is_parity_running; then
        echo "Parity in progress — restart actions skipped ($(date '+%H:%M:%S'))"
        exit 0
    fi

    # ── RAM emergency check — resource_watchdog.sh managing containers ────────────────────────
    # If resource_watchdog.sh has triggered a hard RAM shutdown, defer all container
    # management this run to prevent undoing the emergency stop and re-pressuring RAM.
    if ! check_resource_watchdog_state; then
        MEM_KB=$(awk '/MemAvailable/ {print $2}' /proc/meminfo)
        MEM_GB=$(( MEM_KB / 1024 / 1024 ))
        warn "RAM emergency active (${MEM_GB}GB free) — resource_watchdog.sh managing containers"
        warn "Deferring all container restart logic this run"
        log "Waiting for RAM to recover above ${RW_RAM_RECOVER_GB:-20}GB before resuming"
        exit 0
    fi

    # ── Startup grace period ──────────────────────────────────────────────────────────────────
    UPTIME_SECONDS=$(awk '{print int($1)}' /proc/uptime)


grep -n "IGNORE_MAP\|ignore_map\[" /boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh | head -20

Find where IGNORE_MAP is checked against containers


620:    declare -A IGNORE_MAP
622:        [[ -n "$c" ]] && IGNORE_MAP["$c"]=1
863:                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
882:                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
909:                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
944:                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
970:                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue


855	        ALL_CONTAINERS=$(timeout "$DOCKER_TIMEOUT" docker ps --format "{{.Names}}" 2>/dev/null)
856
857	        # ── Unhealthy containers ──────────────────────────────────────────────────────────────
858	        if [[ "$WATCHDOG_RESTART_UNHEALTHY" == "true" ]]; then
859	            UNHEALTHY=$(timeout "$DOCKER_TIMEOUT" docker ps \
860	                --filter health=unhealthy --format "{{.Names}}" 2>/dev/null)
861	            while IFS= read -r container; do
862	                [[ -z "$container" ]] && continue
863	                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
864	                is_skipped "$container" && continue
865	                error "$container — Docker HEALTHCHECK unhealthy"
866	                ((T2_WARNINGS++))
867	                result=0
868	                safe_restart "$container" "unhealthy health status" || result=$?
869	                if [[ $result -eq 0 ]]; then
870	                    ((T2_RESTARTS++))
871	                    queue_notify "$container unhealthy on $(hostname) — restarted" "warning"
872	                fi
873	            done <<< "$UNHEALTHY"
874	        fi
875
876	        # ── OOM killed ────────────────────────────────────────────────────────────────────────
877	        # OOMKilled flag persists after restart — track handled containers per-cycle
878	        # to prevent the same container triggering a restart loop every cycle
879	        if [[ "$WATCHDOG_NOTIFY_OOM" == "true" ]]; then
880	            while IFS= read -r container; do
881	                [[ -z "$container" ]] && continue
882	                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
883	                is_skipped "$container" && continue
884	                [[ -n "${OOM_HANDLED[$container]:-}" ]] && continue  # already handled this cycle

        ALL_CONTAINERS=$(timeout "$DOCKER_TIMEOUT" docker ps --format "{{.Names}}" 2>/dev/null)

        # ── Unhealthy containers ──────────────────────────────────────────────────────────────
        if [[ "$WATCHDOG_RESTART_UNHEALTHY" == "true" ]]; then
            UNHEALTHY=$(timeout "$DOCKER_TIMEOUT" docker ps \
                --filter health=unhealthy --format "{{.Names}}" 2>/dev/null)
            while IFS= read -r container; do
                [[ -z "$container" ]] && continue
                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
                is_skipped "$container" && continue
                error "$container — Docker HEALTHCHECK unhealthy"
                ((T2_WARNINGS++))
                result=0
                safe_restart "$container" "unhealthy health status" || result=$?
                if [[ $result -eq 0 ]]; then
                    ((T2_RESTARTS++))
                    queue_notify "$container unhealthy on $(hostname) — restarted" "warning"
                fi
            done <<< "$UNHEALTHY"
        fi

        # ── OOM killed ────────────────────────────────────────────────────────────────────────
        # OOMKilled flag persists after restart — track handled containers per-cycle
        # to prevent the same container triggering a restart loop every cycle
        if [[ "$WATCHDOG_NOTIFY_OOM" == "true" ]]; then
            while IFS= read -r container; do
                [[ -z "$container" ]] && continue
                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
                is_skipped "$container" && continue
                [[ -n "${OOM_HANDLED[$container]:-}" ]] && continue  # already handled this cycle


900
901	        # ── Crash loop detection ──────────────────────────────────────────────────────────────
902	        # Tracks Docker's own RestartCount climbing between cycles.
903	        # Below WATCHDOG_CRASH_LIMIT: notify only — Docker's restart policy is handling it.
904	        # At or above WATCHDOG_CRASH_LIMIT: safe_restart() which will add to skip list
905	        # if WATCHDOG_CONTAINER_RESTART_LIMIT is also hit — ensures eventual quarantine.
906	        if [[ "$WATCHDOG_NOTIFY_CRASHLOOP" == "true" ]]; then
907	            while IFS= read -r container; do
908	                [[ -z "$container" ]] && continue
909	                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
910	                is_skipped "$container" && continue
911	                RESTART_COUNT=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
912	                    '{{.RestartCount}}' "$container" 2>/dev/null || echo 0)
913	                PREV_COUNT=$(grep "^${container}_docker:" "$WATCHDOG_STATE_FILE" \
914	                    2>/dev/null | cut -d: -f2 || echo 0)
915	                set_strikes "${container}_docker" "$RESTART_COUNT" "$WATCHDOG_STATE_FILE"
916	                if [[ "$RESTART_COUNT" -gt "$PREV_COUNT" && "$RESTART_COUNT" -gt 0 ]]; then
917	                    ((T2_WARNINGS++))
918	                    if [[ "$RESTART_COUNT" -ge "$WATCHDOG_CRASH_LIMIT" ]]; then
919	                        error "$container — crash loop CRITICAL: $RESTART_COUNT restarts"
920	                        # Attempt restart via safe_restart — will add to skip list if over limit
921	                        result=0
922	                        safe_restart "$container" "crash loop — $RESTART_COUNT restarts" || result=$?
923	                        case $result in
924	                            0) ((T2_RESTARTS++))
925	                               queue_notify "$container crash loop on $(hostname) — restarted" "critical" ;;
926	                            2) : ;; # Added to skip list
927	                            *) queue_notify "$container crash loop CRITICAL on $(hostname) — manual intervention needed" "critical" ;;
928	                        esac
929	                    else
930	                        warn "$container — restarted since last check (Docker count: $RESTART_COUNT)"
931	                        queue_notify "$container restarted on $(hostname) — Docker count: $RESTART_COUNT" "warning"
932	                    fi
933	                fi
934	            done <<< "$ALL_CONTAINERS"
935	        fi
936
937	        # ── Dead containers ───────────────────────────────────────────────────────────────────
938	        # Routes through safe_restart() — ensures restart loop protection applies
939	        if [[ "$WATCHDOG_RESTART_DEAD" == "true" ]]; then
940	            DEAD=$(timeout "$DOCKER_TIMEOUT" docker ps -a \
941	                --filter status=dead --format "{{.Names}}" 2>/dev/null)
942	            while IFS= read -r container; do
943	                [[ -z "$container" ]] && continue
944	                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
945	                is_skipped "$container" && continue
946	                error "$container — dead"
947	                ((T2_WARNINGS++))
948	                if [[ "$DRY_RUN" == false ]]; then
949	                    timeout "$DOCKER_TIMEOUT" docker rm "$container" >/dev/null 2>&1
950	                fi
951	                result=0
952	                safe_restart "$container" "dead container" || result=$?
953	                if [[ $result -eq 0 ]]; then
954	                    ((T2_RESTARTS++))
955	                    queue_notify "$container was dead on $(hostname) — removed and restarted" "warning"
956	                fi
957	            done <<< "$DEAD"
958	        fi
959
960	        # ── Unexpected exits ──────────────────────────────────────────────────────────────────
961	        # Only non-zero exit codes — exit 0 is a clean stop, not a crash
962	        # Skips containers already covered by WATCHDOG_REQUIRED_CONTAINERS (handled in Tier 1)
963	        if [[ "$WATCHDOG_RESTART_CRASHED" == "true" ]]; then
964	            CRASHED=$(timeout "$DOCKER_TIMEOUT" docker ps -a \
965	                --filter status=exited \
966	                --format "{{.Names}}|{{.Status}}" 2>/dev/null | \
967	                grep -v "Exited (0)")
968	            while IFS='|' read -r container status; do
969	                [[ -z "$container" ]] && continue
970	                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
971	                is_skipped "$container" && continue
972	                # Skip containers already monitored by required containers (Tier 1)
973	                already_required=false
974	                for req in "${WATCHDOG_REQUIRED_CONTAINERS[@]:-}"; do
975	                    [[ "$container" == "$req" ]] && already_required=true && break
976	                done
977	                [[ "$already_required" == true ]] && continue
978	                error "$container — $status (unexpected exit)"
979	                ((T2_WARNINGS++))

        # ── Crash loop detection ──────────────────────────────────────────────────────────────
        # Tracks Docker's own RestartCount climbing between cycles.
        # Below WATCHDOG_CRASH_LIMIT: notify only — Docker's restart policy is handling it.
        # At or above WATCHDOG_CRASH_LIMIT: safe_restart() which will add to skip list
        # if WATCHDOG_CONTAINER_RESTART_LIMIT is also hit — ensures eventual quarantine.
        if [[ "$WATCHDOG_NOTIFY_CRASHLOOP" == "true" ]]; then
            while IFS= read -r container; do
                [[ -z "$container" ]] && continue
                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
                is_skipped "$container" && continue
                RESTART_COUNT=$(timeout "$DOCKER_TIMEOUT" docker inspect -f \
                    '{{.RestartCount}}' "$container" 2>/dev/null || echo 0)
                PREV_COUNT=$(grep "^${container}_docker:" "$WATCHDOG_STATE_FILE" \
                    2>/dev/null | cut -d: -f2 || echo 0)
                set_strikes "${container}_docker" "$RESTART_COUNT" "$WATCHDOG_STATE_FILE"
                if [[ "$RESTART_COUNT" -gt "$PREV_COUNT" && "$RESTART_COUNT" -gt 0 ]]; then
                    ((T2_WARNINGS++))
                    if [[ "$RESTART_COUNT" -ge "$WATCHDOG_CRASH_LIMIT" ]]; then
                        error "$container — crash loop CRITICAL: $RESTART_COUNT restarts"
                        # Attempt restart via safe_restart — will add to skip list if over limit
                        result=0
                        safe_restart "$container" "crash loop — $RESTART_COUNT restarts" || result=$?
                        case $result in
                            0) ((T2_RESTARTS++))
                               queue_notify "$container crash loop on $(hostname) — restarted" "critical" ;;
                            2) : ;; # Added to skip list
                            *) queue_notify "$container crash loop CRITICAL on $(hostname) — manual intervention needed" "critical" ;;
                        esac
                    else
                        warn "$container — restarted since last check (Docker count: $RESTART_COUNT)"
                        queue_notify "$container restarted on $(hostname) — Docker count: $RESTART_COUNT" "warning"
                    fi
                fi
            done <<< "$ALL_CONTAINERS"
        fi

        # ── Dead containers ───────────────────────────────────────────────────────────────────
        # Routes through safe_restart() — ensures restart loop protection applies
        if [[ "$WATCHDOG_RESTART_DEAD" == "true" ]]; then
            DEAD=$(timeout "$DOCKER_TIMEOUT" docker ps -a \
                --filter status=dead --format "{{.Names}}" 2>/dev/null)
            while IFS= read -r container; do
                [[ -z "$container" ]] && continue
                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
                is_skipped "$container" && continue
                error "$container — dead"
                ((T2_WARNINGS++))
                if [[ "$DRY_RUN" == false ]]; then
                    timeout "$DOCKER_TIMEOUT" docker rm "$container" >/dev/null 2>&1
                fi
                result=0
                safe_restart "$container" "dead container" || result=$?
                if [[ $result -eq 0 ]]; then
                    ((T2_RESTARTS++))
                    queue_notify "$container was dead on $(hostname) — removed and restarted" "warning"
                fi
            done <<< "$DEAD"
        fi

        # ── Unexpected exits ──────────────────────────────────────────────────────────────────
        # Only non-zero exit codes — exit 0 is a clean stop, not a crash
        # Skips containers already covered by WATCHDOG_REQUIRED_CONTAINERS (handled in Tier 1)
        if [[ "$WATCHDOG_RESTART_CRASHED" == "true" ]]; then
            CRASHED=$(timeout "$DOCKER_TIMEOUT" docker ps -a \
                --filter status=exited \
                --format "{{.Names}}|{{.Status}}" 2>/dev/null | \
                grep -v "Exited (0)")
            while IFS='|' read -r container status; do
                [[ -z "$container" ]] && continue
                [[ -n "${IGNORE_MAP[$container]:-}" ]] && continue
                is_skipped "$container" && continue
                # Skip containers already monitored by required containers (Tier 1)
                already_required=false
                for req in "${WATCHDOG_REQUIRED_CONTAINERS[@]:-}"; do
                    [[ "$container" == "$req" ]] && already_required=true && break
                done
                [[ "$already_required" == true ]] && continue
                error "$container — $status (unexpected exit)"
                ((T2_WARNINGS++))


Check current tail of watchdog orchestrator log


━━━ ⚡ Stability Watchdog — 2026-05-31 20:41:17 ━━━
System healthy ✅ (20:41:18)

━━━ ⚙ Resource Manager — HOST1 — 20:42:01 ━━━
  RAM: 52GB free   Load: 21.50   Action: 0 → 0 (normal)
System at normal pressure ✅
━━━ 🐾 Docker Watchdog — 2026-05-31 20:42:01 ━━━
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
⏱      Duration: 6s
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
━━━ 🛡 System Watchdog — 2026-05-31 20:42:07 ━━━
━━━ 💾 Storage Watchdog — 2026-05-31 20:42:07 ━━━
Storage healthy ✅ (20:42:14)
WebGUI responding — healthy ✅
━━━ 🌐 Network Watchdog — 2026-05-31 20:42:14 ━━━
Network healthy ✅ (20:42:15)
━━━ ⚡ Stability Watchdog — 2026-05-31 20:42:15 ━━━
System healthy ✅ (20:42:15)

━━━ ⚙ Resource Manager — HOST1 — 20:43:01 ━━━
  RAM: 52GB free   Load: 14.23   Action: 0 → 0 (normal)
System at normal pressure ✅
━━━ 🐾 Docker Watchdog — 2026-05-31 20:43:01 ━━━
❌ [ERROR] emby-test — Exited (127) 13 hours ago (unexpected exit)
❌ [ERROR] Failed to restart emby-test (timeout or error)
🐾 T1: 0 restarts / 0 warnings
🐾 T2: 0 restarts / 1 warnings
⏱      Duration: 6s
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
━━━ 🛡 System Watchdog — 2026-05-31 20:43:07 ━━━
━━━ 💾 Storage Watchdog — 2026-05-31 20:43:07 ━━━
Storage healthy ✅ (20:43:14)
WebGUI responding — healthy ✅
━━━ 🌐 Network Watchdog — 2026-05-31 20:43:14 ━━━
Network healthy ✅ (20:43:14)
━━━ ⚡ Stability Watchdog — 2026-05-31 20:43:14 ━━━
System healthy ✅ (20:43:15)


The log is 7 days old — `clear_logs.sh` this morning must have stopped new writes somehow. Let me check the watchdog heartbeat and whether clear_logs broke the log path.


cat /boot/config/plugins/varaverk/State_Files/watchdog_orch_hb.count; stat /boot/config/plugins/varaverk/State_Files/watchdog_orchestrator.log | grep -E "Size|Modify"

Check watchdog heartbeat count and log file modification time


687
  Size: 83773     	Blocks: 17         IO Block: 83968  regular file
Modify: 2026-05-31 20:43:15.401101500 -0400


grep -n "watchdog_orchestrator\|watchdog.*log\|orch.*log" /boot/config/plugins/varaverk/Tools/clear_logs.sh 2>/dev/null | head -20

See how clear_logs handles the watchdog orchestrator log


(Bash completed with no output)


grep -n "watchdog_orchestrator\|orch_hb\|STATE_DIR\|WATCHDOG_LOG" /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh | head -20

Find where watchdog_orchestrator.log path is defined


45:#   watchdog_orchestrator.sh           — normal run (called by cron every 15 minutes)
46:#   watchdog_orchestrator.sh --dry-run — pass --dry-run to all sub-scripts
47:#   watchdog_orchestrator.sh --status  — show script paths and current grace state
48:#   watchdog_orchestrator.sh --log     — verbose output from all sub-scripts
188:    HB_COUNT_FILE="${STATE_DIR:-/tmp}/watchdog_orch_hb.count"
196:        warn "♥ watchdog_orchestrator alive — $MY_ID — ~${HB_HR}hr ($(date '+%H:%M:%S'))"


grep -n "WATCHDOG_LOG\|LOG_FILE\|orchestrator.log\|tee\|exec.*log" /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh | head -20

Find log file setup in watchdog_orchestrator


grep -n "orchestrator.log" /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh /boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh /boot/config/plugins/varaverk/common.sh 2>/dev/null | head -20

Find who writes to watchdog_orchestrator.log


grep -rn "watchdog_orchestrator.log" /boot/config/plugins/varaverk/ 2>/dev/null

Find all references to watchdog_orchestrator.log


grep -rn "WATCHDOG_ORCH_LOG\|orch.*\.log\|LOG.*orch" /boot/config/plugins/varaverk/ 2>/dev/null | head -10

Find watchdog orchestrator log variable definition


grep -n "LOG\|log" /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh | grep -v "^[0-9]*:#\|BLOG\|CHANGELOG\|log_level\|logfile_path" | head -20

Find log references in watchdog_orchestrator


71:log "$ICON_GEAR Config: grace=${WATCHDOG_STARTUP_GRACE}s heartbeat=${WATCHDOG_ORCHESTRATOR_HEARTBEAT:-true}/${WATCHDOG_ORCHESTRATOR_HEARTBEAT_HOURS:-1}hr scripts=${#WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"
72:log "$ICON_WATCHDOG Order: $(for s in "${WATCHDOG_ORCHESTRATOR_SCRIPTS[@]}"; do printf '%s ' "${s##*/}"; done)"
128:log "$ICON_DISK Array: /mnt/user mounted (shfs) ✅"
138:log "Startup grace: past — uptime $(format_duration $UPTIME_SECONDS)"
160:    [[ "$VERBOSE" == true ]] && extra_args+=("--log")
164:    log "$ICON_START $name"
166:        log "$ICON_DONE $name — done in $(format_duration $(( $(date +%s) - _ws )))"
206:    for p in "${PASS[@]}"; do log "  $ICON_DONE  $p"; done


head -30 /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh | grep -v "^#"

Check script header for log setup


grep -n "exec\|tee\|VV_LIVE\|VV_LAST\|LOG_FILE" /boot/config/plugins/varaverk/Orchestrators/watchdog_orchestrator.sh | head -20

Find exec/tee log setup in watchdog orchestrator


grep -n "exec\|tee\|log" /boot/config/plugins/varaverk/Configurations/master.conf | grep -i "watchdog.*log\|orch.*log\|log.*watchdog" | head -10

Find watchdog log config in master.conf


325:        "Watchdogs/System/storage_watchdog.sh"   # pool growth + runaway log detection
762:#   Any container growing more than WATCHDOG_APPDATA_GROWTH_GB triggers a focused *.log scan
767:#   Finds *.log / *.log.* files over WATCHDOG_APPDATA_LOG_MAX_GB across all appdata paths.
774:#     If WATCHDOG_APPDATA_TRUNCATE_LOGS=true: truncate *.log files in-place, clear strikes
784:    WATCHDOG_APPDATA_LOG_MAX_GB=2           # flag *.log files exceeding this size (absolute)
785:    WATCHDOG_APPDATA_TRUNCATE_LOGS=false    # set true to auto-truncate oversized *.log files on action cycle


cat /etc/cron.d/varaverk 2>/dev/null || crontab -l 2>/dev/null | grep -i watchdog

Check cron entry for watchdog orchestrator


cat /etc/cron.d/varaverk 2>/dev/null; ls /etc/cron.d/ 2>/dev/null; cat /var/spool/cron/crontabs/root 2>/dev/null | head -20

Find the actual cron definition for varaverk watchdog


root
# If you don't want the output of a cron job mailed to you, you have to direct
# any output to /dev/null.  We'll do this here since these jobs should run
# properly on a newly installed system.  If a script fails, run-parts will
# mail a notice to root.
#
# Run the hourly, daily, weekly, and monthly cron jobs.
# Jobs that need different timing may be entered into the crontab as before,
# but most really don't need greater granularity than this.  If the exact
# times of the hourly, daily, weekly, and monthly cron jobs do not suit your
# needs, feel free to adjust them.
#
# Run hourly cron jobs at 47 minutes after the hour:
47 * * * * /usr/bin/run-parts /etc/cron.hourly 1> /dev/null
#
# Run daily cron jobs at 4:40 every day:
40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null
#
# Run weekly cron jobs at 4:30 on the first day of the week:
30 4 * * 0 /usr/bin/run-parts /etc/cron.weekly 1> /dev/null
#


grep -rn "watchdog_orchestrator" /boot/config/plugins/varaverk/ 2>/dev/null | grep -v "^.*\.conf\|^.*\.bak" | head -10

Find what invokes watchdog_orchestrator


/boot/config/plugins/varaverk/Watchdogs/resource_watchdog.sh:9:# things break. Called by watchdog_orchestrator.sh every 15 minutes as a
/boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh:9:# watchdog_orchestrator.sh via cron every 15 minutes as a single-pass run.
/boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh:206:#     Single-pass monitoring cycle — called by watchdog_orchestrator.sh every 15 min
/boot/config/plugins/varaverk/Watchdogs/docker_watchdog.sh:281:    echo "$ICON_WATCHDOG   Schedule:       every 15 min (cron via watchdog_orchestrator)"
/boot/config/plugins/varaverk/CLAUDE.md:85:| Every minute | `watchdog_orchestrator.sh` → resource → docker → system → stability watchdogs |
/boot/config/plugins/varaverk/Manual.md:319:| `watchdog_orchestrator` | `*/15 * * * *` | resource → docker → system → stability |
/boot/config/plugins/varaverk/Manual.md:502:bash Orchestrators/watchdog_orchestrator.sh --log
/boot/config/plugins/varaverk/Watchdogs/Manual-Watchdogs.md:5:For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
/boot/config/plugins/varaverk/Watchdogs/Manual-Watchdogs.md:106:resource_watchdog.sh            # single pass (called by watchdog_orchestrator.sh)
/boot/config/plugins/varaverk/Watchdogs/Manual-Watchdogs.md:264:docker_watchdog.sh           # single pass (called by watchdog_orchestrator.sh)


grep -rn "watchdog_orchestrator" /boot/config/go /boot/config/plugins/*.plg /boot/config/plugins/varaverk/*.plg 2>/dev/null | head -10; find /etc/cron.d /var/spool/cron -type f 2>/dev/null | xargs grep -l "watchdog" 2>/dev/null

Find cron/startup entry for watchdog_orchestrator


/boot/config/plugins/varaverk.plg:210:pkill -f "watchdog_orchestrator.sh" 2>/dev/null || true
/etc/cron.d/root


grep "watchdog\|varaverk" /etc/cron.d/root 2>/dev/null

Check root cron for watchdog entries