Varaverk/Notes_AI-Design.md

Varaverk AI Integration — Design Notes

Status: retrieval is built; integration is not. As of 2026-08-02 the AI_* and HOST*_OLLAMA_* variables exist in both confs and both templates, and AI/ holds a working index and query path — see the RAG section at the end of this document and AI/README-AI.md.

Everything else below remains design only. No Varaverk script consults AI. Every AI_ASSIST_* toggle is false, AI_CONF_WRITE_ENABLED is false with an empty whitelist, and host resolution across the mesh is specified but not implemented. Originally captured 2026-08-01 so the reasoning survives.

Ollama itself is installed, tuned and verified on HOST1 — qwen2.5-coder:14b for generation, nomic-embed-text for embeddings, 16k context, pinned to the RTX 3080. See Hardware Budget for measured numbers. That is the substrate, not the integration.

Origin: the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the nvidia driver, not vfio — not reserved for passthrough, so there is no VM contention to design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can genuinely use.

Build order — deliberately lowest-risk first. Each stage must be boring before the next one starts:

1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing
2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated
3. Cleanup and sync decision aid — closest to destructive, last to be trusted

The regular script is always the backup, at every stage.

---

The governing principle

Varaverk works exactly as well with AI off as with it on.

Every script — including scripts written after this lands — is designed and hardened without AI first. AI is added afterwards as enhancement, never as a dependency. A script that cannot do its job when AI_ENABLED=false is a broken script, not an AI feature.

This is the constraint everything else in this document answers to. If a design decision makes AI load-bearing, that decision is wrong.

Corollary: AI never makes a destructive decision. The session that produced the current safeguard layer (depth guards, strike thresholds, verification-after-write) exists because config values and scan results feed rm -rf, chown -R, and rsync --delete. AI advises at the points where a script currently stops and defers to a human. The deterministic guard still pulls the trigger.

---

Two independent off-switches

switch	meaning	source
AI_ENABLED	intent — do we want AI at all	master.conf
resolver result	availability — is there a reachable host	runtime probe

Both must produce the identical code path when off. A caller that gets "no AI" from either must run its normal, non-AI logic — not a degraded variant, not a skipped step.

AI_ENABLED follows the fail-closed idiom standardised across the ecosystem:

[[ "${AI_ENABLED:-false}" != "true" ]] && <normal path>

Not == false. Anything that isn't exactly true means off, so a typo can never switch AI on. (This is the same bug that was fixed in fallback.sh — see its FALLBACK_ENABLED gate.)

---

Configuration schema

Follows the existing rule: thresholds and toggles → master.conf; hardware, paths, container names and per-host identity → host*.conf.

host*.conf — per-host, because only one node actually has the GPU

# ━━━ Ollama / AI ━━━
    HOST1_OLLAMA_URL="http://localhost:11434"   # empty on nodes without a local Ollama
    HOST1_OLLAMA_CONTAINER="Ollama"             # for docker_watchdog / restart lists
    HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626"
    HOST1_OLLAMA_MODEL="qwen2.5-coder:14b"      # generation
    HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed

A node with an empty OLLAMA_URL is not an error — it falls through to the resolver and uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials it is already waiting on.

master.conf — shared behaviour

# ━━━ AI ━━━
    AI_ENABLED=false                # master switch — fail-closed, != "true" means off
    AI_CONNECT_TIMEOUT=5            # probe timeout when resolving a host
    AI_REQUEST_TIMEOUT=240          # must clear a cold load — measured 1m45s after tuning.
                                    # KEEP_ALIVE=-1 means this only bites after a restart,
                                    # but a first call that times out is the worst first
                                    # impression a caller can have. Re-measure against a
                                    # real RAG query before fixing this number.
    AI_RESOLVE_CACHE_TTL=300        # don't re-probe the mesh on every script invocation
    AI_MAX_RETRIES=1                # AI is enhancement — do not retry hard

    # Per-feature toggles — enable narration long before enabling decision aid
    AI_ASSIST_REPORTS=false         # tier 1 — digest / coffee report narration
    AI_ASSIST_WATCHDOG=false        # tier 2 — context on a flagged condition
    AI_ASSIST_DISCOVERY=false       # tier 2 — discovery / classification judgment calls
    AI_ASSIST_CLEANUP=false         # tier 2 — HELD orphans, stuck-import triage
    AI_ASSIST_ONBOARD=false         # tier 3 — onboarding / settings assistance

    # Conf writes — separate switch, off by default, see Conf Write Access below
    AI_CONF_WRITE_ENABLED=false
    AI_CONF_WRITE_KEYS=()           # explicit whitelist; never paths or credentials

Per-feature toggles are load-bearing, not decoration. They are what lets AI narrate the weekly digest for months before it is ever allowed near a cleanup decision. AI_ENABLED is necessary but not sufficient — every feature stays individually off until it has earned it.

When these land, Deployment/master.conf.template and Deployment/host.conf.template must be updated in the same pass. That rule is not optional in this repo.

---

Host resolution

AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node needs a GPU, a model, or an Ollama container — only the resolver.

resolve_ollama_host() mirrors the existing Gitea locator in git_pull_execute.sh:

Ollama answering on localhost:11434?      → use it            (owner's node)
  else discover_remote_nodes()
       → resolve_tailscale_ip(node)
       → probe each :11434                → first responsive wins
  else                                    → no AI host        (== AI_ENABLED=false)

Helpers already exist in common.sh: discover_remote_nodes() (767), resolve_tailscale_ip() (812), check_connectivity() (866).

Probe the API, not the container. The Gitea locator checks docker ps. Do not copy that here — a container can be up while the model is unloaded, still pulling, or wedged. Probe /api/tags. Same principle written into network_watchdog.sh's design principles: verify the path, not the process.

Cache the resolution in /tmp state, like the arr cache. A cleanup script should not pay a Tailscale round-trip to discover AI it may never call.

Known consequence

AI lives on HOST1, so during a fallback — HOST1 down, HOST2 covering — the mesh has no AI. That is precisely when a triage assistant would be most useful. Accepted: a second GPU on HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth knowing rather than discovering.

---

Where AI is allowed to act

Ranked by how much damage a wrong answer does.

Tier 1 — narration and summary (safe, do first)

• Sunday morning coffee report — turn metrics into prose
• weekly_health_digest.sh — summarise, highlight what changed
• Explain why a container is crash-looping from its logs

Tier 2 — triage and context on an existing flag (the real value) Places where a script already detects something and stops:

• system_watchdog / stability_watchdog flags a condition → AI adds context, correlates with recent logs, suggests likely cause
• Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work
• HELD entries from arr_download_orphan_cleaner.sh
• reverse-anime-leak from the classification scans — currently report-only because it is a judgment call. That is exactly the shape AI suits.

Tier 3 — assisted configuration (needs the guardrails below)

• Onboarding a new host — the main motivation for conf write access
• AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack

Never

• Deciding what to delete
• Choosing a path for any destructive operation
• Anything that bypasses a strike counter, age gate, or verification step

---

Conf write access

Wanted mainly for onboarding and assisted settings. This is the highest-risk item here.

Current state: there is no recovery path.

.gitignore:4  Configurations/host*.conf
.gitignore:5  Configurations/master.conf
.gitignore:6  Configurations/*.bak

Confs are gitignored — no git history to revert to — and so are the .bak files, so the backup is not versioned either. The only fallback is a single .bak slot written by conf_upgrade, and it goes stale immediately:

file	modified	its .bak
master.conf	Jul 28 18:52	Jul 28 18:52
host1.conf	Aug 1 21:00	Jul 3 17:46

A bad write to master.conf currently falls back to a file that may predate a month of edits. Fix this before any AI writes anything.

Required before conf-write ships

1. Key whitelist, not file access. Thresholds and toggles only — *_WARN_GB, *_STRIKE_LIMIT, *_ENABLED, retention days. Never a path, never a credential, never a container list. A wrong threshold is recoverable; a wrong path is what the depth guards exist to catch.
2. Timestamped backups, plural — master.conf.2026-08-01T21:00, retained. Not one clobbered slot.
3. Validate before commit — bash -n the candidate, then confirm load_config.sh sources it cleanly. Never install a conf that has not been proven to parse.
4. Diff always logged. An AI conf change should be at least as visible as a container restart.
5. Lock against concurrent readers — never rewrite a conf while scripts are mid-run.

Consider un-ignoring Configurations/ into a private repo. Then git diff and git revert become the recovery mechanism and the history is free. This overlaps the existing GitHub-mirror TODO, which is already blocked on the same question.

---

Security

Ollama has no authentication of any kind, and its API includes DELETE /api/delete (wipe models) and POST /api/pull (fill the disk). It currently binds 0.0.0.0:11434 with OLLAMA_ORIGINS=* — reachable from the entire LAN, not just Tailscale.

The design only needs loopback (owner) plus the Tailscale interface (mesh). 0.0.0.0 is strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than app-level auth Ollama cannot provide anyway.

Done 2026-08-01: /ext-varaverk is now mounted ro (was rw into live prod). Verified on the running container — rw=false.

Still open: the LAN exposure above. Deliberately not folded into the tuning rebuild, since bind-address versus Tailscale ACL is a decision rather than a setting.

---

Hardware budget

RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby transcodes on. Do not let AI onto the P2000.

Tuned and measured 2026-08-01. These are observed values, not estimates.

	before	after
OLLAMA_NUM_PARALLEL	2	1
OLLAMA_KV_CACHE_TYPE	f16	q8_0
OLLAMA_CONTEXT_LENGTH	4096	16384
OLLAMA_FLASH_ATTENTION	false	true
VRAM used	9298 MiB (91%)	8811 MiB (86%)
warm latency	2.6s	1.85s
cold load	28.6s	1m45s

/api/ps confirms ctx=16384 — the increase is real, not just an env var.

4× the context for less VRAM than before. Flash Attention plus the quantized KV cache more than paid for the increase. Cold load got much slower, which is irrelevant while OLLAMA_KEEP_ALIVE=-1 pins both models — but it is felt after any container restart.

Flash Attention is mandatory, not optional

OLLAMA_KV_CACHE_TYPE=q8_0 will not load without it:

llama_init_from_model: V cache quantization requires flash_attn
llama-server process no longer running: exit status 1

Quantized K/V cache requires Flash Attention. Supported on Ampere and newer; the 3080 qualifies. If the KV cache type is ever changed back toward a quantized value, Flash Attention must be on or the model silently fails to load and every call errors.

Tuning order still matters

If these are ever re-tuned from defaults, the order is load-bearing — raising context first at high utilisation will OOM:

1. OLLAMA_NUM_PARALLEL → 1 (each slot multiplies KV cache)
2. OLLAMA_FLASH_ATTENTION → true (prerequisite for the next step)
3. OLLAMA_KV_CACHE_TYPE → q8_0 (roughly halves KV memory)
4. then OLLAMA_CONTEXT_LENGTH upward

32k was considered and rejected — projected ~1000 MiB of KV, leaving under 450 MiB headroom. 16k is the comfortable ceiling for a 14B on this card.

Concurrency

You cannot have 14B + long context + real parallelism on 10 GB. Pick two. OLLAMA_MAX_QUEUE=512 means excess requests queue rather than fail, and at ~2s responses, two or three users serialised is barely noticeable. Multi-user hits are expected to be rare. So: parallelism stays at 1, the queue absorbs bursts, and the VRAM goes to context — which is what RAG actually needs.

Applying template changes

Unraid's "Apply" does not reliably recreate the container. Observed 2026-08-01: the template was saved correctly but the container was only restarted, so the env vars never took effect — Created stayed unchanged while Started advanced. Env changes require a remove-and-recreate.

Force it with Unraid's own script:

/usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container Ollama
docker start Ollama      # rebuild stops it — Ollama is not in unraid-autostart

Verify with docker inspect Ollama --format '{{.Created}}' — the timestamp must move. Checking the env vars alone is not enough; a restart leaves the old ones in place and looks like nothing happened.

---

RAG

No Python on Unraid, and none needed. Everything required is already present: sqlite3 3.53, jq 1.8, node 22, php 8.4, awk.

• chunks + vectors → SQLite, one table
• similarity → cosine in PHP or Node; milliseconds at this corpus size, no vector DB container needed
• embed + generate → Ollama HTTP, same curl pattern as every other integration here

Note qwen2.5-coder returns 501 — does not support embeddings. Embedding is nomic-embed-text's job. Batch embedding works (n inputs → n vectors in one call) and is required — indexing 50k lines one HTTP call at a time is not viable.

---

The corpus — and why its shape matters more than its size

As of 2026-08-01, after the header audit and the per-folder documentation pass:

Layer	Size	What it answers
Script headers	115 files × 6 sections = 690 chunks, 14,685 lines	"What does this script do, and why that way"
Folder docs	18 README-*.md + 13 Manual-*.md	"How does this group work" / "how do I do the thing"
Top-level	README.md, Manual.md	"What is this system"
Conf templates	2,175 lines, ~55% comment	The schema, self-describing
Bash bodies	51,166 lines	Implementation — index last, lowest weight

Markdown total: 13,516 lines. Still small enough that cosine over the whole set is milliseconds.

Chunking is already solved, and the audit is what solved it. Every script carries PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION / RUNTIME MODES — 115 of 115, no exceptions. Split on ^# SECTION NAME$ and every chunk is a semantically coherent unit by construction. The single worst failure mode in naive RAG — a fixed-size window cutting mid-thought and embedding two half-ideas as one vector — cannot happen here. Median header is 117 lines, so a section lands around 130–200 tokens: comfortably inside nomic-embed-text's window, no sub-splitting needed.

Store the section name as a column, not just as chunk text. This is the highest-value thing the audit bought and it should not be thrown away at index time. Section type is a free metadata filter, so retrieval can route before it computes similarity:

Question shape	Filter to
"what stops X and Y overlapping"	OPERATIONAL SAFEGUARDS
"what variable controls X"	CONFIGURATION
"does this take --dry-run"	RUNTIME MODES
"why is it built this way"	DESIGN PRINCIPLES
"what does this script do"	PURPOSE

Hybrid retrieval essentially for free, because every chunk already has a type.

Suggested table shape:

CREATE TABLE vv_chunks (
    id        INTEGER PRIMARY KEY,
    path      TEXT NOT NULL,     -- repo-relative
    kind      TEXT NOT NULL,     -- header | readme | manual | template | body
    section   TEXT,              -- PURPOSE, OPERATIONAL SAFEGUARDS, ... (NULL for md/body)
    heading   TEXT,              -- md ## heading, for doc chunks
    content   TEXT NOT NULL,
    vector    BLOB NOT NULL,     -- 768 float32
    indexed   INTEGER NOT NULL   -- epoch; re-embed on mtime change only
);

Why this corpus is worth more than an equivalent pile of code

A model can read mover_stop.sh and describe what it does. What it cannot derive from any amount of source is that a thing was done deliberately. The audit wrote those down:

• the API cache writers are lockless and unprivileged on purpose — regenerable within a minute, every consumer has a live fallback
• removeCompletedDownloads / removeFailedDownloads both true is intended, not an oversight
• the arr cleanup ctime gate depends on media_shares_permissions.sh staying conditional — reverting either silently stops orphan collection
• mesh_monitor.sh, adapter.sh, decision_engine.sh, containers.sh and api_cache_writer.sh carry no root check and no lock by design — each documents why in its own header (libraries that must not exit, read-only probes, or regenerable output with a live fallback)

Without those in the index, the most likely contribution from an AI assistant reviewing this repo is a confident regression: "I notice this script lacks a lock." Weight DESIGN PRINCIPLES and OPERATIONAL SAFEGUARDS heavily for any suggest-a-change flow — they are the guardrails against the assistant helpfully undoing a decision.

Indexing is safe by default — keep it that way

Configurations/*.conf is gitignored; Deployment/*.template is tracked and carries all the explanatory comments. The corpus therefore describes the full schema while structurally never containing a credential, because the credential-bearing files were never in the repo to begin with.

Treat that as a deliberate boundary, not a happy accident:

• index tracked files only — never walk Configurations/, State_Files/, or data/
• a live conf value that the model genuinely needs should arrive through a tool call at query time, subject to the same redaction rules as everything else in the Security section, not be baked into a vector at index time
• an embedded secret is unrevocable in a way a logged one is not — there is no rotation story for a value already averaged into a 768-dim float

Known gap — the PHP layer is not covered

78 PHP files under Plugin/unraid/; 2 carry a PURPOSE block. The entire web UI — pages/, api/, include/ — is effectively invisible to retrieval.

Consequence: any "AI helper per Varaverk page" feature has this as a hard prerequisite. A page-scoped assistant that cannot retrieve the page's own logic is worse than no assistant.

include/ is the high-value subset to do first — 16 files, and both the pages and the API endpoints route through the same vv_*() builders, so documenting it once covers both callers. This is a follow-on pass, not a blocker for indexing bash.

---

Scheduled AI

Same orchestrator tiers as everything else, gated on AI_ENABLED plus a reachable host. Natural fits: weekly digest narration, a periodic pass over HELD/report-only findings that have accumulated, post-incident summaries after a watchdog event.

Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal step like any other, and never blocks the rest of its tier.

---

UI

• Dedicated AI page in the plugin.
• Persistent conversation across pages. A floating widget is not required — a chat column is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no SPA shell, so persistence means conversation state lives server-side keyed by session, with the client re-hydrating per page.
• Per-page AI helpers — contextual assistance scoped to whatever that page is about.

---

"Too bad we can't just run the LLM inside Varaverk and cut out Ollama"

There is a real answer: you don't cut Ollama out, you absorb it.

Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers better than most things manage containers. Ollama becomes just another managed container:

• add to HOST*_WATCHDOG_CONTAINERS so docker_watchdog.sh keeps it healthy
• add to a restart list so it gets the same proactive treatment as everything else
• give it a fallback tier if AI should survive a host outage
• let docker_update.sh handle its image updates

That is more Varaverk-native than embedding a model runtime would be, and it costs nothing new — the machinery already exists and was audited this session.

---

Open questions

• Un-ignore Configurations/ into a private repo for conf history? (blocks conf-write, and overlaps the GitHub-mirror TODO)
• Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members?
• Retention/privacy for conversation history — logs may contain paths, container names, possibly credentials pasted by a user.
• Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the serialisation? Defer until an actual problem is felt.

---

RAG — built 2026-08-02

Retrieval is live. AI/ holds the implementation; AI/README-AI.md documents it in full. What follows is only what changed relative to the plan recorded above.

Corpus is larger than estimated. ~2,950 chunks across ~180 files, not the 690 header chunks projected. Sub-chunking is why — see below.

Named-paragraph sub-chunking was necessary, and was not in the plan. Section-level chunks alone were too coarse. rsync.sh documents fourteen safeguards in one 2.8k-char OPERATIONAL SAFEGUARDS block; a query about one of them scored 0.558, below unrelated chunks, because the other thirteen dominated the vector. Splitting on the named-paragraph titles the header convention already uses took the same query to 0.718 and first place. The parent section name is carried onto each sub-chunk so routing still works.

Two chunker bugs worth remembering. The last section in a header (RUNTIME MODES in bash, DEPENDS ON in a page) ran to EOF and swept up every unrelated comment in the file — scheduler.php alone produced an 11k-char chunk of unrelated inline comments. And title detection must require the next line to be indented; without that, any wrapped prose line became a spurious chunk boundary mid-sentence.

Section routing is a boost, not a filter. Intent detection is a heuristic and must not be able to exclude the chunk holding the answer. --section= forces a hard filter when wanted.

Vectors arrive pre-normalised. nomic-embed-text returns L2-normalised vectors (measured norm 1.0000001), so cosine is a plain dot product. No normalising step, no magnitude cache.

node:sqlite over a native module. Still flagged experimental, chosen because it needs no native compilation on Unraid. Acceptable because the index is disposable — if a Node upgrade breaks it, rebuild takes minutes. PHP reads the same float32 blobs with unpack('f*', $blob) when the UI needs them.

Retrieval quality, measured. 9/10 top-3 hit rate on known-answer questions; the tenth had the answer at ranks 2 and 3, so 10/10 for answer-present-in-context at k=6. Full retrieval plus generation runs about 43s warm.

The finding that validated the whole thing

First real end-to-end question asked which variable controls the mover's grace window. The model answered MOVER_STOP_TIMEOUT, "defaults to 30 seconds", citing mover_stop.sh › CONFIGURATION. Variable correct; the 30 was wrong — the real value is 300. The model was quoting the header verbatim. The header was stale.

A sweep for the same pattern found six stale (default: N) claims across the repo — all corrected in the same pass. This is the operating principle for the folder:

Retrieval is exactly as accurate as the documentation it points at. When an answer looks wrong, check the cited source before blaming the model.

It also means the index is a documentation-drift detector, not only a question-answering tool.

Still deliberately not built

Nothing consults this. Every AI_ASSIST_* toggle is false, AI_CONF_WRITE_ENABLED is false with an empty key whitelist, and no watchdog, cleanup or fallback path calls it. Host resolution across the Tailscale mesh is specified above but not implemented — ai_index.sh and ai_query.sh currently require a local HOST*_OLLAMA_URL and fail with a clear message when it is empty, rather than silently probing the mesh.