# Varaverk AI Integration — Design Notes **Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no `AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives. Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See Hardware Budget for measured numbers. That is the substrate, not the integration. **Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the `nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can genuinely use. **Build order** — deliberately lowest-risk first. Each stage must be boring before the next one starts: 1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing 2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated 3. Cleanup and sync decision aid — closest to destructive, last to be trusted The regular script is always the backup, at every stage. --- ## The governing principle **Varaverk works exactly as well with AI off as with it on.** Every script — including scripts written after this lands — is designed and hardened without AI first. AI is added afterwards as enhancement, never as a dependency. A script that cannot do its job when `AI_ENABLED=false` is a broken script, not an AI feature. This is the constraint everything else in this document answers to. If a design decision makes AI load-bearing, that decision is wrong. Corollary: AI never makes a destructive decision. The session that produced the current safeguard layer (depth guards, strike thresholds, verification-after-write) exists because config values and scan results feed `rm -rf`, `chown -R`, and rsync `--delete`. AI advises at the points where a script currently stops and defers to a human. The deterministic guard still pulls the trigger. --- ## Two independent off-switches | switch | meaning | source | |---|---|---| | `AI_ENABLED` | intent — do we want AI at all | `master.conf` | | resolver result | availability — is there a reachable host | runtime probe | **Both must produce the identical code path when off.** A caller that gets "no AI" from either must run its normal, non-AI logic — not a degraded variant, not a skipped step. `AI_ENABLED` follows the fail-closed idiom standardised across the ecosystem: ```bash [[ "${AI_ENABLED:-false}" != "true" ]] && ``` Not `== false`. Anything that isn't exactly `true` means off, so a typo can never switch AI on. (This is the same bug that was fixed in `fallback.sh` — see its `FALLBACK_ENABLED` gate.) --- ## Configuration schema Follows the existing rule: **thresholds and toggles → `master.conf`; hardware, paths, container names and per-host identity → `host*.conf`.** ### `host*.conf` — per-host, because only one node actually has the GPU ```bash # ━━━ Ollama / AI ━━━ HOST1_OLLAMA_URL="http://localhost:11434" # empty on nodes without a local Ollama HOST1_OLLAMA_CONTAINER="Ollama" # for docker_watchdog / restart lists HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626" HOST1_OLLAMA_MODEL="qwen2.5-coder:14b" # generation HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed ``` A node with an empty `OLLAMA_URL` is not an error — it falls through to the resolver and uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials it is already waiting on. ### `master.conf` — shared behaviour ```bash # ━━━ AI ━━━ AI_ENABLED=false # master switch — fail-closed, != "true" means off AI_CONNECT_TIMEOUT=5 # probe timeout when resolving a host AI_REQUEST_TIMEOUT=240 # must clear a cold load — measured 1m45s after tuning. # KEEP_ALIVE=-1 means this only bites after a restart, # but a first call that times out is the worst first # impression a caller can have. Re-measure against a # real RAG query before fixing this number. AI_RESOLVE_CACHE_TTL=300 # don't re-probe the mesh on every script invocation AI_MAX_RETRIES=1 # AI is enhancement — do not retry hard # Per-feature toggles — enable narration long before enabling decision aid AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgment calls AI_ASSIST_CLEANUP=false # tier 2 — HELD orphans, stuck-import triage AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance # Conf writes — separate switch, off by default, see Conf Write Access below AI_CONF_WRITE_ENABLED=false AI_CONF_WRITE_KEYS=() # explicit whitelist; never paths or credentials ``` **Per-feature toggles are load-bearing, not decoration.** They are what lets AI narrate the weekly digest for months before it is ever allowed near a cleanup decision. `AI_ENABLED` is necessary but not sufficient — every feature stays individually off until it has earned it. > When these land, `Deployment/master.conf.template` and `Deployment/host.conf.template` > must be updated in the same pass. That rule is not optional in this repo. --- ## Host resolution AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node needs a GPU, a model, or an Ollama container — only the resolver. `resolve_ollama_host()` mirrors the existing Gitea locator in `git_pull_execute.sh`: ``` Ollama answering on localhost:11434? → use it (owner's node) else discover_remote_nodes() → resolve_tailscale_ip(node) → probe each :11434 → first responsive wins else → no AI host (== AI_ENABLED=false) ``` Helpers already exist in `common.sh`: `discover_remote_nodes()` (767), `resolve_tailscale_ip()` (812), `check_connectivity()` (866). **Probe the API, not the container.** The Gitea locator checks `docker ps`. Do not copy that here — a container can be up while the model is unloaded, still pulling, or wedged. Probe `/api/tags`. Same principle written into `network_watchdog.sh`'s design principles: *verify the path, not the process*. **Cache the resolution** in `/tmp` state, like the arr cache. A cleanup script should not pay a Tailscale round-trip to discover AI it may never call. ### Known consequence AI lives on HOST1, so **during a fallback — HOST1 down, HOST2 covering — the mesh has no AI.** That is precisely when a triage assistant would be most useful. Accepted: a second GPU on HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth knowing rather than discovering. --- ## Where AI is allowed to act Ranked by how much damage a wrong answer does. **Tier 1 — narration and summary (safe, do first)** - Sunday morning coffee report — turn metrics into prose - `weekly_health_digest.sh` — summarise, highlight what changed - Explain *why* a container is crash-looping from its logs **Tier 2 — triage and context on an existing flag (the real value)** Places where a script already detects something and stops: - `system_watchdog` / `stability_watchdog` flags a condition → AI adds context, correlates with recent logs, suggests likely cause - Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work - `HELD` entries from `arr_download_orphan_cleaner.sh` - `reverse-anime-leak` from the classification scans — currently report-only *because* it is a judgment call. That is exactly the shape AI suits. **Tier 3 — assisted configuration (needs the guardrails below)** - Onboarding a new host — the main motivation for conf write access - AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack **Never** - Deciding what to delete - Choosing a path for any destructive operation - Anything that bypasses a strike counter, age gate, or verification step --- ## Conf write access Wanted mainly for onboarding and assisted settings. This is the highest-risk item here. **Current state: there is no recovery path.** ``` .gitignore:4 Configurations/host*.conf .gitignore:5 Configurations/master.conf .gitignore:6 Configurations/*.bak ``` Confs are gitignored — no git history to revert to — and so are the `.bak` files, so the backup is not versioned either. The only fallback is a single `.bak` slot written by `conf_upgrade`, and it goes stale immediately: | file | modified | its `.bak` | |---|---|---| | `master.conf` | Jul 28 18:52 | Jul 28 18:52 | | `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** | A bad write to `master.conf` currently falls back to a file that may predate a month of edits. **Fix this before any AI writes anything.** ### Required before conf-write ships 1. **Key whitelist, not file access.** Thresholds and toggles only — `*_WARN_GB`, `*_STRIKE_LIMIT`, `*_ENABLED`, retention days. Never a path, never a credential, never a container list. A wrong threshold is recoverable; a wrong path is what the depth guards exist to catch. 2. **Timestamped backups, plural** — `master.conf.2026-08-01T21:00`, retained. Not one clobbered slot. 3. **Validate before commit** — `bash -n` the candidate, then confirm `load_config.sh` sources it cleanly. Never install a conf that has not been proven to parse. 4. **Diff always logged.** An AI conf change should be at least as visible as a container restart. 5. **Lock against concurrent readers** — never rewrite a conf while scripts are mid-run. **Consider un-ignoring `Configurations/` into a private repo.** Then `git diff` and `git revert` become the recovery mechanism and the history is free. This overlaps the existing GitHub-mirror TODO, which is already blocked on the same question. --- ## Security Ollama has **no authentication of any kind**, and its API includes `DELETE /api/delete` (wipe models) and `POST /api/pull` (fill the disk). It currently binds `0.0.0.0:11434` with `OLLAMA_ORIGINS=*` — reachable from the entire LAN, not just Tailscale. The design only needs loopback (owner) plus the Tailscale interface (mesh). `0.0.0.0` is strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than app-level auth Ollama cannot provide anyway. **Done 2026-08-01:** `/ext-varaverk` is now mounted `ro` (was `rw` into live prod). Verified on the running container — `rw=false`. **Still open:** the LAN exposure above. Deliberately not folded into the tuning rebuild, since bind-address versus Tailscale ACL is a decision rather than a setting. --- ## Hardware budget RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby transcodes on. Do not let AI onto the P2000. **Tuned and measured 2026-08-01.** These are observed values, not estimates. | | before | after | |---|---|---| | `OLLAMA_NUM_PARALLEL` | 2 | **1** | | `OLLAMA_KV_CACHE_TYPE` | f16 | **q8_0** | | `OLLAMA_CONTEXT_LENGTH` | 4096 | **16384** | | `OLLAMA_FLASH_ATTENTION` | false | **true** | | VRAM used | 9298 MiB (91%) | **8811 MiB (86%)** | | warm latency | 2.6s | **1.85s** | | cold load | 28.6s | **1m45s** | `/api/ps` confirms `ctx=16384` — the increase is real, not just an env var. **4× the context for less VRAM than before.** Flash Attention plus the quantized KV cache more than paid for the increase. Cold load got much slower, which is irrelevant while `OLLAMA_KEEP_ALIVE=-1` pins both models — but it is felt after any container restart. ### Flash Attention is mandatory, not optional `OLLAMA_KV_CACHE_TYPE=q8_0` **will not load** without it: ``` llama_init_from_model: V cache quantization requires flash_attn llama-server process no longer running: exit status 1 ``` Quantized K/V cache requires Flash Attention. Supported on Ampere and newer; the 3080 qualifies. If the KV cache type is ever changed back toward a quantized value, Flash Attention must be on or the model silently fails to load and every call errors. ### Tuning order still matters If these are ever re-tuned from defaults, the order is load-bearing — raising context first at high utilisation will OOM: 1. `OLLAMA_NUM_PARALLEL` → 1 (each slot multiplies KV cache) 2. `OLLAMA_FLASH_ATTENTION` → true (prerequisite for the next step) 3. `OLLAMA_KV_CACHE_TYPE` → q8_0 (roughly halves KV memory) 4. *then* `OLLAMA_CONTEXT_LENGTH` upward 32k was considered and rejected — projected ~1000 MiB of KV, leaving under 450 MiB headroom. 16k is the comfortable ceiling for a 14B on this card. ### Concurrency **You cannot have 14B + long context + real parallelism on 10 GB.** Pick two. `OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~2s responses, two or three users serialised is barely noticeable. Multi-user hits are expected to be rare. So: parallelism stays at 1, the queue absorbs bursts, and the VRAM goes to **context** — which is what RAG actually needs. ### Applying template changes **Unraid's "Apply" does not reliably recreate the container.** Observed 2026-08-01: the template was saved correctly but the container was only *restarted*, so the env vars never took effect — `Created` stayed unchanged while `Started` advanced. Env changes require a remove-and-recreate. Force it with Unraid's own script: ```bash /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container Ollama docker start Ollama # rebuild stops it — Ollama is not in unraid-autostart ``` Verify with `docker inspect Ollama --format '{{.Created}}'` — the timestamp must move. Checking the env vars alone is not enough; a restart leaves the old ones in place and looks like nothing happened. --- ## RAG No Python on Unraid, and none needed. Everything required is already present: `sqlite3 3.53`, `jq 1.8`, `node 22`, `php 8.4`, `awk`. - **chunks + vectors** → SQLite, one table - **similarity** → cosine in PHP or Node; milliseconds at this corpus size, no vector DB container needed - **embed + generate** → Ollama HTTP, same `curl` pattern as every other integration here Note `qwen2.5-coder` returns `501 — does not support embeddings`. Embedding is `nomic-embed-text`'s job. Batch embedding works (n inputs → n vectors in one call) and is required — indexing 50k lines one HTTP call at a time is not viable. --- ### The corpus — and why its shape matters more than its size As of 2026-08-01, after the header audit and the per-folder documentation pass: | Layer | Size | What it answers | |-------|------|-----------------| | Script headers | 115 files × 6 sections = **690 chunks**, 14,685 lines | "What does *this script* do, and why that way" | | Folder docs | 18 `README-*.md` + 13 `Manual-*.md` | "How does this *group* work" / "how do I do the thing" | | Top-level | `README.md`, `Manual.md` | "What is this system" | | Conf templates | 2,175 lines, **~55% comment** | The schema, self-describing | | Bash bodies | 51,166 lines | Implementation — index last, lowest weight | Markdown total: **13,516 lines.** Still small enough that cosine over the whole set is milliseconds. **Chunking is already solved, and the audit is what solved it.** Every script carries `PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION / RUNTIME MODES` — **115 of 115, no exceptions.** Split on `^# SECTION NAME$` and every chunk is a semantically coherent unit by construction. The single worst failure mode in naive RAG — a fixed-size window cutting mid-thought and embedding two half-ideas as one vector — cannot happen here. Median header is 117 lines, so a section lands around 130–200 tokens: comfortably inside `nomic-embed-text`'s window, no sub-splitting needed. **Store the section name as a column, not just as chunk text.** This is the highest-value thing the audit bought and it should not be thrown away at index time. Section type is a free metadata filter, so retrieval can route before it computes similarity: | Question shape | Filter to | |----------------|-----------| | "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` | | "what variable controls X" | `CONFIGURATION` | | "does this take --dry-run" | `RUNTIME MODES` | | "why is it built this way" | `DESIGN PRINCIPLES` | | "what does this script do" | `PURPOSE` | Hybrid retrieval essentially for free, because every chunk already has a type. Suggested table shape: ```sql CREATE TABLE vv_chunks ( id INTEGER PRIMARY KEY, path TEXT NOT NULL, -- repo-relative kind TEXT NOT NULL, -- header | readme | manual | template | body section TEXT, -- PURPOSE, OPERATIONAL SAFEGUARDS, ... (NULL for md/body) heading TEXT, -- md ## heading, for doc chunks content TEXT NOT NULL, vector BLOB NOT NULL, -- 768 float32 indexed INTEGER NOT NULL -- epoch; re-embed on mtime change only ); ``` ### Why this corpus is worth more than an equivalent pile of code A model can read `mover_stop.sh` and describe what it does. What it *cannot* derive from any amount of source is that a thing was done deliberately. The audit wrote those down: - the API cache writers are lockless and unprivileged **on purpose** — regenerable within a minute, every consumer has a live fallback - `removeCompletedDownloads` / `removeFailedDownloads` both true is **intended**, not an oversight - the arr cleanup ctime gate depends on `media_shares_permissions.sh` staying conditional — reverting either silently stops orphan collection - `mesh_monitor.sh`, `adapter.sh`, `decision_engine.sh`, `containers.sh` and `api_cache_writer.sh` carry no root check and no lock **by design** — each documents why in its own header (libraries that must not `exit`, read-only probes, or regenerable output with a live fallback) Without those in the index, the most likely contribution from an AI assistant reviewing this repo is a confident regression: *"I notice this script lacks a lock."* Weight `DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS` heavily for any suggest-a-change flow — they are the guardrails against the assistant helpfully undoing a decision. ### Indexing is safe by default — keep it that way `Configurations/*.conf` is gitignored; `Deployment/*.template` is tracked and carries all the explanatory comments. The corpus therefore describes the full schema while structurally **never containing a credential**, because the credential-bearing files were never in the repo to begin with. Treat that as a deliberate boundary, not a happy accident: - **index tracked files only** — never walk `Configurations/`, `State_Files/`, or `data/` - a live conf value that the model genuinely needs should arrive through a *tool call* at query time, subject to the same redaction rules as everything else in the Security section, not be baked into a vector at index time - an embedded secret is unrevocable in a way a logged one is not — there is no rotation story for a value already averaged into a 768-dim float ### Known gap — the PHP layer is not covered 78 PHP files under `Plugin/unraid/`; **2** carry a `PURPOSE` block. The entire web UI — `pages/`, `api/`, `include/` — is effectively invisible to retrieval. Consequence: any "AI helper per Varaverk page" feature has this as a hard prerequisite. A page-scoped assistant that cannot retrieve the page's own logic is worse than no assistant. `include/` is the high-value subset to do first — 16 files, and both the pages and the API endpoints route through the same `vv_*()` builders, so documenting it once covers both callers. This is a follow-on pass, not a blocker for indexing bash. --- ## Scheduled AI Same orchestrator tiers as everything else, gated on `AI_ENABLED` plus a reachable host. Natural fits: weekly digest narration, a periodic pass over `HELD`/report-only findings that have accumulated, post-incident summaries after a watchdog event. Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal step like any other, and never blocks the rest of its tier. --- ## UI - **Dedicated AI page** in the plugin. - **Persistent conversation across pages.** A floating widget is not required — a chat column is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no SPA shell, so persistence means conversation state lives server-side keyed by session, with the client re-hydrating per page. - **Per-page AI helpers** — contextual assistance scoped to whatever that page is about. --- ## "Too bad we can't just run the LLM inside Varaverk and cut out Ollama" There is a real answer: you don't cut Ollama out, you **absorb it**. Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers better than most things manage containers. Ollama becomes just another managed container: - add to `HOST*_WATCHDOG_CONTAINERS` so `docker_watchdog.sh` keeps it healthy - add to a restart list so it gets the same proactive treatment as everything else - give it a fallback tier if AI should survive a host outage - let `docker_update.sh` handle its image updates That is more Varaverk-native than embedding a model runtime would be, and it costs nothing new — the machinery already exists and was audited this session. --- ## Open questions - Un-ignore `Configurations/` into a private repo for conf history? (blocks conf-write, and overlaps the GitHub-mirror TODO) - Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members? - Retention/privacy for conversation history — logs may contain paths, container names, possibly credentials pasted by a user. - Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the serialisation? Defer until an actual problem is felt.