From b80cf04cdc2dc03f109d5a334c794769160003aa Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 1 Aug 2026 21:56:02 -0400 Subject: [PATCH] Record measured Ollama tuning results in the AI design notes The estimates in the original note were wrong in both directions, and Unraid's Apply does not reliably recreate a container, which is worth knowing before the next tuning pass. --- Notes_AI-Design.md | 99 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 78 insertions(+), 21 deletions(-) diff --git a/Notes_AI-Design.md b/Notes_AI-Design.md index a497418..08cfb92 100644 --- a/Notes_AI-Design.md +++ b/Notes_AI-Design.md @@ -1,7 +1,11 @@ # Varaverk AI Integration — Design Notes -**Status: design only. Nothing below is built.** Ollama is installed and running on HOST1; -no Varaverk script calls it. Captured 2026-08-01 so the reasoning survives. +**Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no +`AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives. + +Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for +generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See +Hardware Budget for measured numbers. That is the substrate, not the integration. **Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the `nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to @@ -85,7 +89,11 @@ it is already waiting on. # ━━━ AI ━━━ AI_ENABLED=false # master switch — fail-closed, != "true" means off AI_CONNECT_TIMEOUT=5 # probe timeout when resolving a host - AI_REQUEST_TIMEOUT=120 # generation can be slow; cold load was 28.6s + AI_REQUEST_TIMEOUT=240 # must clear a cold load — measured 1m45s after tuning. + # KEEP_ALIVE=-1 means this only bites after a restart, + # but a first call that times out is the worst first + # impression a caller can have. Re-measure against a + # real RAG query before fixing this number. AI_RESOLVE_CACHE_TTL=300 # don't re-probe the mesh on every script invocation AI_MAX_RETRIES=1 # AI is enhancement — do not retry hard @@ -229,8 +237,11 @@ strictly wider than required, for no benefit. Restrict to loopback + Tailscale, Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than app-level auth Ollama cannot provide anyway. -Also set: `/ext-varaverk` mount is now `ro` in the template (was `rw` into live prod). -**Requires an Apply in the Docker tab to take effect on the running container.** +**Done 2026-08-01:** `/ext-varaverk` is now mounted `ro` (was `rw` into live prod). +Verified on the running container — `rw=false`. + +**Still open:** the LAN exposure above. Deliberately not folded into the tuning rebuild, +since bind-address versus Tailscale ACL is a decision rather than a setting. --- @@ -239,29 +250,75 @@ Also set: `/ext-varaverk` mount is now `ro` in the template (was `rw` into live RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby transcodes on. Do not let AI onto the P2000. -| | VRAM | -|---|---| -| `qwen2.5-coder:14b` (Q4_K_M) | 8.34 GB | -| `nomic-embed-text` (768-dim) | 0.25 GB | -| **used / total** | **9.30 / 10.24 GB (91%)** | +**Tuned and measured 2026-08-01.** These are observed values, not estimates. -Latency: 28.6s cold load, ~2.6s warm, ~4.4s warm with both models resident. -`OLLAMA_KEEP_ALIVE=-1` keeps both pinned, so cold load is a boot-time cost only. +| | before | after | +|---|---|---| +| `OLLAMA_NUM_PARALLEL` | 2 | **1** | +| `OLLAMA_KV_CACHE_TYPE` | f16 | **q8_0** | +| `OLLAMA_CONTEXT_LENGTH` | 4096 | **16384** | +| `OLLAMA_FLASH_ATTENTION` | false | **true** | +| VRAM used | 9298 MiB (91%) | **8811 MiB (86%)** | +| warm latency | 2.6s | **1.85s** | +| cold load | 28.6s | **1m45s** | + +`/api/ps` confirms `ctx=16384` — the increase is real, not just an env var. + +**4× the context for less VRAM than before.** Flash Attention plus the quantized KV cache +more than paid for the increase. Cold load got much slower, which is irrelevant while +`OLLAMA_KEEP_ALIVE=-1` pins both models — but it is felt after any container restart. + +### Flash Attention is mandatory, not optional + +`OLLAMA_KV_CACHE_TYPE=q8_0` **will not load** without it: + +``` +llama_init_from_model: V cache quantization requires flash_attn +llama-server process no longer running: exit status 1 +``` + +Quantized K/V cache requires Flash Attention. Supported on Ampere and newer; the 3080 +qualifies. If the KV cache type is ever changed back toward a quantized value, Flash +Attention must be on or the model silently fails to load and every call errors. + +### Tuning order still matters + +If these are ever re-tuned from defaults, the order is load-bearing — raising context first +at high utilisation will OOM: + +1. `OLLAMA_NUM_PARALLEL` → 1 (each slot multiplies KV cache) +2. `OLLAMA_FLASH_ATTENTION` → true (prerequisite for the next step) +3. `OLLAMA_KV_CACHE_TYPE` → q8_0 (roughly halves KV memory) +4. *then* `OLLAMA_CONTEXT_LENGTH` upward + +32k was considered and rejected — projected ~1000 MiB of KV, leaving under 450 MiB headroom. +16k is the comfortable ceiling for a 14B on this card. + +### Concurrency **You cannot have 14B + long context + real parallelism on 10 GB.** Pick two. -`OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~4s responses, +`OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~2s responses, two or three users serialised is barely noticeable. Multi-user hits are expected to be rare. -So: keep parallelism low, let the queue absorb bursts, spend VRAM on **context** — that is -what RAG actually needs. +So: parallelism stays at 1, the queue absorbs bursts, and the VRAM goes to **context** — +which is what RAG actually needs. -Tuning order matters. At 91% utilisation, raising context first will OOM: +### Applying template changes -1. `OLLAMA_NUM_PARALLEL` 2 → 1 (frees KV cache) -2. `OLLAMA_KV_CACHE_TYPE` f16 → q8_0 (roughly halves KV memory, negligible quality cost) -3. *then* `OLLAMA_CONTEXT_LENGTH` 4096 → 8192/16384 +**Unraid's "Apply" does not reliably recreate the container.** Observed 2026-08-01: the +template was saved correctly but the container was only *restarted*, so the env vars never +took effect — `Created` stayed unchanged while `Started` advanced. Env changes require a +remove-and-recreate. -All three are env vars in the same template, so they ride along with the same Apply as the -`ro` mount fix — one container recreate covers everything. +Force it with Unraid's own script: + +```bash +/usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container Ollama +docker start Ollama # rebuild stops it — Ollama is not in unraid-autostart +``` + +Verify with `docker inspect Ollama --format '{{.Created}}'` — the timestamp must move. +Checking the env vars alone is not enough; a restart leaves the old ones in place and +looks like nothing happened. ---