# Varaverk AI Integration — Design Notes **Status: design only. Nothing below is built.** Ollama is installed and running on HOST1; no Varaverk script calls it. Captured 2026-08-01 so the reasoning survives. **Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the `nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can genuinely use. **Build order** — deliberately lowest-risk first. Each stage must be boring before the next one starts: 1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing 2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated 3. Cleanup and sync decision aid — closest to destructive, last to be trusted The regular script is always the backup, at every stage. --- ## The governing principle **Varaverk works exactly as well with AI off as with it on.** Every script — including scripts written after this lands — is designed and hardened without AI first. AI is added afterwards as enhancement, never as a dependency. A script that cannot do its job when `AI_ENABLED=false` is a broken script, not an AI feature. This is the constraint everything else in this document answers to. If a design decision makes AI load-bearing, that decision is wrong. Corollary: AI never makes a destructive decision. The session that produced the current safeguard layer (depth guards, strike thresholds, verification-after-write) exists because config values and scan results feed `rm -rf`, `chown -R`, and rsync `--delete`. AI advises at the points where a script currently stops and defers to a human. The deterministic guard still pulls the trigger. --- ## Two independent off-switches | switch | meaning | source | |---|---|---| | `AI_ENABLED` | intent — do we want AI at all | `master.conf` | | resolver result | availability — is there a reachable host | runtime probe | **Both must produce the identical code path when off.** A caller that gets "no AI" from either must run its normal, non-AI logic — not a degraded variant, not a skipped step. `AI_ENABLED` follows the fail-closed idiom standardised across the ecosystem: ```bash [[ "${AI_ENABLED:-false}" != "true" ]] && ``` Not `== false`. Anything that isn't exactly `true` means off, so a typo can never switch AI on. (This is the same bug that was fixed in `fallback.sh` — see its `FALLBACK_ENABLED` gate.) --- ## Configuration schema Follows the existing rule: **thresholds and toggles → `master.conf`; hardware, paths, container names and per-host identity → `host*.conf`.** ### `host*.conf` — per-host, because only one node actually has the GPU ```bash # ━━━ Ollama / AI ━━━ HOST1_OLLAMA_URL="http://localhost:11434" # empty on nodes without a local Ollama HOST1_OLLAMA_CONTAINER="Ollama" # for docker_watchdog / restart lists HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626" HOST1_OLLAMA_MODEL="qwen2.5-coder:14b" # generation HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed ``` A node with an empty `OLLAMA_URL` is not an error — it falls through to the resolver and uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials it is already waiting on. ### `master.conf` — shared behaviour ```bash # ━━━ AI ━━━ AI_ENABLED=false # master switch — fail-closed, != "true" means off AI_CONNECT_TIMEOUT=5 # probe timeout when resolving a host AI_REQUEST_TIMEOUT=120 # generation can be slow; cold load was 28.6s AI_RESOLVE_CACHE_TTL=300 # don't re-probe the mesh on every script invocation AI_MAX_RETRIES=1 # AI is enhancement — do not retry hard # Per-feature toggles — enable narration long before enabling decision aid AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgment calls AI_ASSIST_CLEANUP=false # tier 2 — HELD orphans, stuck-import triage AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance # Conf writes — separate switch, off by default, see Conf Write Access below AI_CONF_WRITE_ENABLED=false AI_CONF_WRITE_KEYS=() # explicit whitelist; never paths or credentials ``` **Per-feature toggles are load-bearing, not decoration.** They are what lets AI narrate the weekly digest for months before it is ever allowed near a cleanup decision. `AI_ENABLED` is necessary but not sufficient — every feature stays individually off until it has earned it. > When these land, `Deployment/master.conf.template` and `Deployment/host.conf.template` > must be updated in the same pass. That rule is not optional in this repo. --- ## Host resolution AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node needs a GPU, a model, or an Ollama container — only the resolver. `resolve_ollama_host()` mirrors the existing Gitea locator in `git_pull_execute.sh`: ``` Ollama answering on localhost:11434? → use it (owner's node) else discover_remote_nodes() → resolve_tailscale_ip(node) → probe each :11434 → first responsive wins else → no AI host (== AI_ENABLED=false) ``` Helpers already exist in `common.sh`: `discover_remote_nodes()` (767), `resolve_tailscale_ip()` (812), `check_connectivity()` (866). **Probe the API, not the container.** The Gitea locator checks `docker ps`. Do not copy that here — a container can be up while the model is unloaded, still pulling, or wedged. Probe `/api/tags`. Same principle written into `network_watchdog.sh`'s design principles: *verify the path, not the process*. **Cache the resolution** in `/tmp` state, like the arr cache. A cleanup script should not pay a Tailscale round-trip to discover AI it may never call. ### Known consequence AI lives on HOST1, so **during a fallback — HOST1 down, HOST2 covering — the mesh has no AI.** That is precisely when a triage assistant would be most useful. Accepted: a second GPU on HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth knowing rather than discovering. --- ## Where AI is allowed to act Ranked by how much damage a wrong answer does. **Tier 1 — narration and summary (safe, do first)** - Sunday morning coffee report — turn metrics into prose - `weekly_health_digest.sh` — summarise, highlight what changed - Explain *why* a container is crash-looping from its logs **Tier 2 — triage and context on an existing flag (the real value)** Places where a script already detects something and stops: - `system_watchdog` / `stability_watchdog` flags a condition → AI adds context, correlates with recent logs, suggests likely cause - Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work - `HELD` entries from `arr_download_orphan_cleaner.sh` - `reverse-anime-leak` from the classification scans — currently report-only *because* it is a judgment call. That is exactly the shape AI suits. **Tier 3 — assisted configuration (needs the guardrails below)** - Onboarding a new host — the main motivation for conf write access - AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack **Never** - Deciding what to delete - Choosing a path for any destructive operation - Anything that bypasses a strike counter, age gate, or verification step --- ## Conf write access Wanted mainly for onboarding and assisted settings. This is the highest-risk item here. **Current state: there is no recovery path.** ``` .gitignore:4 Configurations/host*.conf .gitignore:5 Configurations/master.conf .gitignore:6 Configurations/*.bak ``` Confs are gitignored — no git history to revert to — and so are the `.bak` files, so the backup is not versioned either. The only fallback is a single `.bak` slot written by `conf_upgrade`, and it goes stale immediately: | file | modified | its `.bak` | |---|---|---| | `master.conf` | Jul 28 18:52 | Jul 28 18:52 | | `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** | A bad write to `master.conf` currently falls back to a file that may predate a month of edits. **Fix this before any AI writes anything.** ### Required before conf-write ships 1. **Key whitelist, not file access.** Thresholds and toggles only — `*_WARN_GB`, `*_STRIKE_LIMIT`, `*_ENABLED`, retention days. Never a path, never a credential, never a container list. A wrong threshold is recoverable; a wrong path is what the depth guards exist to catch. 2. **Timestamped backups, plural** — `master.conf.2026-08-01T21:00`, retained. Not one clobbered slot. 3. **Validate before commit** — `bash -n` the candidate, then confirm `load_config.sh` sources it cleanly. Never install a conf that has not been proven to parse. 4. **Diff always logged.** An AI conf change should be at least as visible as a container restart. 5. **Lock against concurrent readers** — never rewrite a conf while scripts are mid-run. **Consider un-ignoring `Configurations/` into a private repo.** Then `git diff` and `git revert` become the recovery mechanism and the history is free. This overlaps the existing GitHub-mirror TODO, which is already blocked on the same question. --- ## Security Ollama has **no authentication of any kind**, and its API includes `DELETE /api/delete` (wipe models) and `POST /api/pull` (fill the disk). It currently binds `0.0.0.0:11434` with `OLLAMA_ORIGINS=*` — reachable from the entire LAN, not just Tailscale. The design only needs loopback (owner) plus the Tailscale interface (mesh). `0.0.0.0` is strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than app-level auth Ollama cannot provide anyway. Also set: `/ext-varaverk` mount is now `ro` in the template (was `rw` into live prod). **Requires an Apply in the Docker tab to take effect on the running container.** --- ## Hardware budget RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby transcodes on. Do not let AI onto the P2000. | | VRAM | |---|---| | `qwen2.5-coder:14b` (Q4_K_M) | 8.34 GB | | `nomic-embed-text` (768-dim) | 0.25 GB | | **used / total** | **9.30 / 10.24 GB (91%)** | Latency: 28.6s cold load, ~2.6s warm, ~4.4s warm with both models resident. `OLLAMA_KEEP_ALIVE=-1` keeps both pinned, so cold load is a boot-time cost only. **You cannot have 14B + long context + real parallelism on 10 GB.** Pick two. `OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~4s responses, two or three users serialised is barely noticeable. Multi-user hits are expected to be rare. So: keep parallelism low, let the queue absorb bursts, spend VRAM on **context** — that is what RAG actually needs. Tuning order matters. At 91% utilisation, raising context first will OOM: 1. `OLLAMA_NUM_PARALLEL` 2 → 1 (frees KV cache) 2. `OLLAMA_KV_CACHE_TYPE` f16 → q8_0 (roughly halves KV memory, negligible quality cost) 3. *then* `OLLAMA_CONTEXT_LENGTH` 4096 → 8192/16384 All three are env vars in the same template, so they ride along with the same Apply as the `ro` mount fix — one container recreate covers everything. --- ## RAG No Python on Unraid, and none needed. Everything required is already present: `sqlite3 3.53`, `jq 1.8`, `node 22`, `php 8.4`, `awk`. - **chunks + vectors** → SQLite, one table - **similarity** → cosine in PHP or Node; milliseconds at this corpus size, no vector DB container needed - **embed + generate** → Ollama HTTP, same `curl` pattern as every other integration here Corpus: **49,654 lines of bash + ~11,970 lines of markdown.** Small. **Chunking is already solved.** Every script now carries `PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION / RUNTIME MODES` at exact, greppable boundaries. Those are semantically coherent units with stable headings — far better retrieval chunks than fixed-size windows. The `DESIGN PRINCIPLES` sections encode *why*, which is what the model needs and what code alone never says. Note `qwen2.5-coder` returns `501 — does not support embeddings`. Embedding is `nomic-embed-text`'s job. Batch embedding works (n inputs → n vectors in one call) and is required — indexing 50k lines one HTTP call at a time is not viable. --- ## Scheduled AI Same orchestrator tiers as everything else, gated on `AI_ENABLED` plus a reachable host. Natural fits: weekly digest narration, a periodic pass over `HELD`/report-only findings that have accumulated, post-incident summaries after a watchdog event. Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal step like any other, and never blocks the rest of its tier. --- ## UI - **Dedicated AI page** in the plugin. - **Persistent conversation across pages.** A floating widget is not required — a chat column is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no SPA shell, so persistence means conversation state lives server-side keyed by session, with the client re-hydrating per page. - **Per-page AI helpers** — contextual assistance scoped to whatever that page is about. --- ## "Too bad we can't just run the LLM inside Varaverk and cut out Ollama" There is a real answer: you don't cut Ollama out, you **absorb it**. Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers better than most things manage containers. Ollama becomes just another managed container: - add to `HOST*_WATCHDOG_CONTAINERS` so `docker_watchdog.sh` keeps it healthy - add to a restart list so it gets the same proactive treatment as everything else - give it a fallback tier if AI should survive a host outage - let `docker_update.sh` handle its image updates That is more Varaverk-native than embedding a model runtime would be, and it costs nothing new — the machinery already exists and was audited this session. --- ## Open questions - Un-ignore `Configurations/` into a private repo for conf history? (blocks conf-write, and overlaps the GitHub-mirror TODO) - Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members? - Retention/privacy for conversation history — logs may contain paths, container names, possibly credentials pasted by a user. - Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the serialisation? Defer until an actual problem is felt.