Add AI integration design notes
Captures the reasoning behind a not-yet-built feature so the constraints survive the session, chiefly that AI stays enhancement-only and never load-bearing.
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,340 @@
|
||||
# Varaverk AI Integration — Design Notes
|
||||
|
||||
**Status: design only. Nothing below is built.** Ollama is installed and running on HOST1;
|
||||
no Varaverk script calls it. Captured 2026-08-01 so the reasoning survives.
|
||||
|
||||
**Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the
|
||||
`nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to
|
||||
design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can
|
||||
genuinely use.
|
||||
|
||||
**Build order** — deliberately lowest-risk first. Each stage must be boring before the next
|
||||
one starts:
|
||||
|
||||
1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing
|
||||
2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated
|
||||
3. Cleanup and sync decision aid — closest to destructive, last to be trusted
|
||||
|
||||
The regular script is always the backup, at every stage.
|
||||
|
||||
---
|
||||
|
||||
## The governing principle
|
||||
|
||||
**Varaverk works exactly as well with AI off as with it on.**
|
||||
|
||||
Every script — including scripts written after this lands — is designed and hardened
|
||||
without AI first. AI is added afterwards as enhancement, never as a dependency. A script
|
||||
that cannot do its job when `AI_ENABLED=false` is a broken script, not an AI feature.
|
||||
|
||||
This is the constraint everything else in this document answers to. If a design decision
|
||||
makes AI load-bearing, that decision is wrong.
|
||||
|
||||
Corollary: AI never makes a destructive decision. The session that produced the current
|
||||
safeguard layer (depth guards, strike thresholds, verification-after-write) exists because
|
||||
config values and scan results feed `rm -rf`, `chown -R`, and rsync `--delete`. AI advises
|
||||
at the points where a script currently stops and defers to a human. The deterministic guard
|
||||
still pulls the trigger.
|
||||
|
||||
---
|
||||
|
||||
## Two independent off-switches
|
||||
|
||||
| switch | meaning | source |
|
||||
|---|---|---|
|
||||
| `AI_ENABLED` | intent — do we want AI at all | `master.conf` |
|
||||
| resolver result | availability — is there a reachable host | runtime probe |
|
||||
|
||||
**Both must produce the identical code path when off.** A caller that gets "no AI" from
|
||||
either must run its normal, non-AI logic — not a degraded variant, not a skipped step.
|
||||
|
||||
`AI_ENABLED` follows the fail-closed idiom standardised across the ecosystem:
|
||||
|
||||
```bash
|
||||
[[ "${AI_ENABLED:-false}" != "true" ]] && <normal path>
|
||||
```
|
||||
|
||||
Not `== false`. Anything that isn't exactly `true` means off, so a typo can never switch AI
|
||||
on. (This is the same bug that was fixed in `fallback.sh` — see its `FALLBACK_ENABLED` gate.)
|
||||
|
||||
---
|
||||
|
||||
## Configuration schema
|
||||
|
||||
Follows the existing rule: **thresholds and toggles → `master.conf`; hardware, paths,
|
||||
container names and per-host identity → `host*.conf`.**
|
||||
|
||||
### `host*.conf` — per-host, because only one node actually has the GPU
|
||||
|
||||
```bash
|
||||
# ━━━ Ollama / AI ━━━
|
||||
HOST1_OLLAMA_URL="http://localhost:11434" # empty on nodes without a local Ollama
|
||||
HOST1_OLLAMA_CONTAINER="Ollama" # for docker_watchdog / restart lists
|
||||
HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626"
|
||||
HOST1_OLLAMA_MODEL="qwen2.5-coder:14b" # generation
|
||||
HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed
|
||||
```
|
||||
|
||||
A node with an empty `OLLAMA_URL` is not an error — it falls through to the resolver and
|
||||
uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials
|
||||
it is already waiting on.
|
||||
|
||||
### `master.conf` — shared behaviour
|
||||
|
||||
```bash
|
||||
# ━━━ AI ━━━
|
||||
AI_ENABLED=false # master switch — fail-closed, != "true" means off
|
||||
AI_CONNECT_TIMEOUT=5 # probe timeout when resolving a host
|
||||
AI_REQUEST_TIMEOUT=120 # generation can be slow; cold load was 28.6s
|
||||
AI_RESOLVE_CACHE_TTL=300 # don't re-probe the mesh on every script invocation
|
||||
AI_MAX_RETRIES=1 # AI is enhancement — do not retry hard
|
||||
|
||||
# Per-feature toggles — enable narration long before enabling decision aid
|
||||
AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration
|
||||
AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition
|
||||
AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgment calls
|
||||
AI_ASSIST_CLEANUP=false # tier 2 — HELD orphans, stuck-import triage
|
||||
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
|
||||
|
||||
# Conf writes — separate switch, off by default, see Conf Write Access below
|
||||
AI_CONF_WRITE_ENABLED=false
|
||||
AI_CONF_WRITE_KEYS=() # explicit whitelist; never paths or credentials
|
||||
```
|
||||
|
||||
**Per-feature toggles are load-bearing, not decoration.** They are what lets AI narrate the
|
||||
weekly digest for months before it is ever allowed near a cleanup decision. `AI_ENABLED` is
|
||||
necessary but not sufficient — every feature stays individually off until it has earned it.
|
||||
|
||||
> When these land, `Deployment/master.conf.template` and `Deployment/host.conf.template`
|
||||
> must be updated in the same pass. That rule is not optional in this repo.
|
||||
|
||||
---
|
||||
|
||||
## Host resolution
|
||||
|
||||
AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node
|
||||
needs a GPU, a model, or an Ollama container — only the resolver.
|
||||
|
||||
`resolve_ollama_host()` mirrors the existing Gitea locator in `git_pull_execute.sh`:
|
||||
|
||||
```
|
||||
Ollama answering on localhost:11434? → use it (owner's node)
|
||||
else discover_remote_nodes()
|
||||
→ resolve_tailscale_ip(node)
|
||||
→ probe each :11434 → first responsive wins
|
||||
else → no AI host (== AI_ENABLED=false)
|
||||
```
|
||||
|
||||
Helpers already exist in `common.sh`: `discover_remote_nodes()` (767),
|
||||
`resolve_tailscale_ip()` (812), `check_connectivity()` (866).
|
||||
|
||||
**Probe the API, not the container.** The Gitea locator checks `docker ps`. Do not copy that
|
||||
here — a container can be up while the model is unloaded, still pulling, or wedged. Probe
|
||||
`/api/tags`. Same principle written into `network_watchdog.sh`'s design principles:
|
||||
*verify the path, not the process*.
|
||||
|
||||
**Cache the resolution** in `/tmp` state, like the arr cache. A cleanup script should not pay
|
||||
a Tailscale round-trip to discover AI it may never call.
|
||||
|
||||
### Known consequence
|
||||
|
||||
AI lives on HOST1, so **during a fallback — HOST1 down, HOST2 covering — the mesh has no AI.**
|
||||
That is precisely when a triage assistant would be most useful. Accepted: a second GPU on
|
||||
HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth
|
||||
knowing rather than discovering.
|
||||
|
||||
---
|
||||
|
||||
## Where AI is allowed to act
|
||||
|
||||
Ranked by how much damage a wrong answer does.
|
||||
|
||||
**Tier 1 — narration and summary (safe, do first)**
|
||||
- Sunday morning coffee report — turn metrics into prose
|
||||
- `weekly_health_digest.sh` — summarise, highlight what changed
|
||||
- Explain *why* a container is crash-looping from its logs
|
||||
|
||||
**Tier 2 — triage and context on an existing flag (the real value)**
|
||||
Places where a script already detects something and stops:
|
||||
- `system_watchdog` / `stability_watchdog` flags a condition → AI adds context, correlates
|
||||
with recent logs, suggests likely cause
|
||||
- Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work
|
||||
- `HELD` entries from `arr_download_orphan_cleaner.sh`
|
||||
- `reverse-anime-leak` from the classification scans — currently report-only *because* it is
|
||||
a judgment call. That is exactly the shape AI suits.
|
||||
|
||||
**Tier 3 — assisted configuration (needs the guardrails below)**
|
||||
- Onboarding a new host — the main motivation for conf write access
|
||||
- AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack
|
||||
|
||||
**Never**
|
||||
- Deciding what to delete
|
||||
- Choosing a path for any destructive operation
|
||||
- Anything that bypasses a strike counter, age gate, or verification step
|
||||
|
||||
---
|
||||
|
||||
## Conf write access
|
||||
|
||||
Wanted mainly for onboarding and assisted settings. This is the highest-risk item here.
|
||||
|
||||
**Current state: there is no recovery path.**
|
||||
|
||||
```
|
||||
.gitignore:4 Configurations/host*.conf
|
||||
.gitignore:5 Configurations/master.conf
|
||||
.gitignore:6 Configurations/*.bak
|
||||
```
|
||||
|
||||
Confs are gitignored — no git history to revert to — and so are the `.bak` files, so the
|
||||
backup is not versioned either. The only fallback is a single `.bak` slot written by
|
||||
`conf_upgrade`, and it goes stale immediately:
|
||||
|
||||
| file | modified | its `.bak` |
|
||||
|---|---|---|
|
||||
| `master.conf` | Jul 28 18:52 | Jul 28 18:52 |
|
||||
| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** |
|
||||
|
||||
A bad write to `master.conf` currently falls back to a file that may predate a month of edits.
|
||||
**Fix this before any AI writes anything.**
|
||||
|
||||
### Required before conf-write ships
|
||||
|
||||
1. **Key whitelist, not file access.** Thresholds and toggles only — `*_WARN_GB`,
|
||||
`*_STRIKE_LIMIT`, `*_ENABLED`, retention days. Never a path, never a credential, never a
|
||||
container list. A wrong threshold is recoverable; a wrong path is what the depth guards
|
||||
exist to catch.
|
||||
2. **Timestamped backups, plural** — `master.conf.2026-08-01T21:00`, retained. Not one
|
||||
clobbered slot.
|
||||
3. **Validate before commit** — `bash -n` the candidate, then confirm `load_config.sh`
|
||||
sources it cleanly. Never install a conf that has not been proven to parse.
|
||||
4. **Diff always logged.** An AI conf change should be at least as visible as a container
|
||||
restart.
|
||||
5. **Lock against concurrent readers** — never rewrite a conf while scripts are mid-run.
|
||||
|
||||
**Consider un-ignoring `Configurations/` into a private repo.** Then `git diff` and
|
||||
`git revert` become the recovery mechanism and the history is free. This overlaps the
|
||||
existing GitHub-mirror TODO, which is already blocked on the same question.
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
Ollama has **no authentication of any kind**, and its API includes `DELETE /api/delete`
|
||||
(wipe models) and `POST /api/pull` (fill the disk). It currently binds `0.0.0.0:11434` with
|
||||
`OLLAMA_ORIGINS=*` — reachable from the entire LAN, not just Tailscale.
|
||||
|
||||
The design only needs loopback (owner) plus the Tailscale interface (mesh). `0.0.0.0` is
|
||||
strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use
|
||||
Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than
|
||||
app-level auth Ollama cannot provide anyway.
|
||||
|
||||
Also set: `/ext-varaverk` mount is now `ro` in the template (was `rw` into live prod).
|
||||
**Requires an Apply in the Docker tab to take effect on the running container.**
|
||||
|
||||
---
|
||||
|
||||
## Hardware budget
|
||||
|
||||
RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby
|
||||
transcodes on. Do not let AI onto the P2000.
|
||||
|
||||
| | VRAM |
|
||||
|---|---|
|
||||
| `qwen2.5-coder:14b` (Q4_K_M) | 8.34 GB |
|
||||
| `nomic-embed-text` (768-dim) | 0.25 GB |
|
||||
| **used / total** | **9.30 / 10.24 GB (91%)** |
|
||||
|
||||
Latency: 28.6s cold load, ~2.6s warm, ~4.4s warm with both models resident.
|
||||
`OLLAMA_KEEP_ALIVE=-1` keeps both pinned, so cold load is a boot-time cost only.
|
||||
|
||||
**You cannot have 14B + long context + real parallelism on 10 GB.** Pick two.
|
||||
`OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~4s responses,
|
||||
two or three users serialised is barely noticeable. Multi-user hits are expected to be rare.
|
||||
So: keep parallelism low, let the queue absorb bursts, spend VRAM on **context** — that is
|
||||
what RAG actually needs.
|
||||
|
||||
Tuning order matters. At 91% utilisation, raising context first will OOM:
|
||||
|
||||
1. `OLLAMA_NUM_PARALLEL` 2 → 1 (frees KV cache)
|
||||
2. `OLLAMA_KV_CACHE_TYPE` f16 → q8_0 (roughly halves KV memory, negligible quality cost)
|
||||
3. *then* `OLLAMA_CONTEXT_LENGTH` 4096 → 8192/16384
|
||||
|
||||
All three are env vars in the same template, so they ride along with the same Apply as the
|
||||
`ro` mount fix — one container recreate covers everything.
|
||||
|
||||
---
|
||||
|
||||
## RAG
|
||||
|
||||
No Python on Unraid, and none needed. Everything required is already present:
|
||||
`sqlite3 3.53`, `jq 1.8`, `node 22`, `php 8.4`, `awk`.
|
||||
|
||||
- **chunks + vectors** → SQLite, one table
|
||||
- **similarity** → cosine in PHP or Node; milliseconds at this corpus size, no vector DB
|
||||
container needed
|
||||
- **embed + generate** → Ollama HTTP, same `curl` pattern as every other integration here
|
||||
|
||||
Corpus: **49,654 lines of bash + ~11,970 lines of markdown.** Small.
|
||||
|
||||
**Chunking is already solved.** Every script now carries
|
||||
`PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION /
|
||||
RUNTIME MODES` at exact, greppable boundaries. Those are semantically coherent units with
|
||||
stable headings — far better retrieval chunks than fixed-size windows. The `DESIGN PRINCIPLES`
|
||||
sections encode *why*, which is what the model needs and what code alone never says.
|
||||
|
||||
Note `qwen2.5-coder` returns `501 — does not support embeddings`. Embedding is
|
||||
`nomic-embed-text`'s job. Batch embedding works (n inputs → n vectors in one call) and is
|
||||
required — indexing 50k lines one HTTP call at a time is not viable.
|
||||
|
||||
---
|
||||
|
||||
## Scheduled AI
|
||||
|
||||
Same orchestrator tiers as everything else, gated on `AI_ENABLED` plus a reachable host.
|
||||
Natural fits: weekly digest narration, a periodic pass over `HELD`/report-only findings that
|
||||
have accumulated, post-incident summaries after a watchdog event.
|
||||
|
||||
Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal
|
||||
step like any other, and never blocks the rest of its tier.
|
||||
|
||||
---
|
||||
|
||||
## UI
|
||||
|
||||
- **Dedicated AI page** in the plugin.
|
||||
- **Persistent conversation across pages.** A floating widget is not required — a chat column
|
||||
is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no
|
||||
SPA shell, so persistence means conversation state lives server-side keyed by session, with
|
||||
the client re-hydrating per page.
|
||||
- **Per-page AI helpers** — contextual assistance scoped to whatever that page is about.
|
||||
|
||||
---
|
||||
|
||||
## "Too bad we can't just run the LLM inside Varaverk and cut out Ollama"
|
||||
|
||||
There is a real answer: you don't cut Ollama out, you **absorb it**.
|
||||
|
||||
Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP
|
||||
API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers
|
||||
better than most things manage containers. Ollama becomes just another managed container:
|
||||
|
||||
- add to `HOST*_WATCHDOG_CONTAINERS` so `docker_watchdog.sh` keeps it healthy
|
||||
- add to a restart list so it gets the same proactive treatment as everything else
|
||||
- give it a fallback tier if AI should survive a host outage
|
||||
- let `docker_update.sh` handle its image updates
|
||||
|
||||
That is more Varaverk-native than embedding a model runtime would be, and it costs nothing
|
||||
new — the machinery already exists and was audited this session.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
- Un-ignore `Configurations/` into a private repo for conf history? (blocks conf-write, and
|
||||
overlaps the GitHub-mirror TODO)
|
||||
- Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members?
|
||||
- Retention/privacy for conversation history — logs may contain paths, container names,
|
||||
possibly credentials pasted by a user.
|
||||
- Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the
|
||||
serialisation? Defer until an actual problem is felt.
|
||||
Reference in New Issue
Block a user