Varaverk/Notes_AI-Design.md

# Varaverk AI Integration — Design Notes

**Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no
`AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives.

Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for
generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See
Hardware Budget for measured numbers. That is the substrate, not the integration.

**Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the
`nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to
design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can
genuinely use.

**Build order** — deliberately lowest-risk first. Each stage must be boring before the next
one starts:

1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing
2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated
3. Cleanup and sync decision aid — closest to destructive, last to be trusted

The regular script is always the backup, at every stage.

---

## The governing principle

**Varaverk works exactly as well with AI off as with it on.**

Every script — including scripts written after this lands — is designed and hardened
without AI first. AI is added afterwards as enhancement, never as a dependency. A script
that cannot do its job when `AI_ENABLED=false` is a broken script, not an AI feature.

This is the constraint everything else in this document answers to. If a design decision
makes AI load-bearing, that decision is wrong.

Corollary: AI never makes a destructive decision. The session that produced the current
safeguard layer (depth guards, strike thresholds, verification-after-write) exists because
config values and scan results feed `rm -rf`, `chown -R`, and rsync `--delete`. AI advises
at the points where a script currently stops and defers to a human. The deterministic guard
still pulls the trigger.

---

## Two independent off-switches

| switch | meaning | source |
|---|---|---|
| `AI_ENABLED` | intent — do we want AI at all | `master.conf` |
| resolver result | availability — is there a reachable host | runtime probe |

**Both must produce the identical code path when off.** A caller that gets "no AI" from
either must run its normal, non-AI logic — not a degraded variant, not a skipped step.

`AI_ENABLED` follows the fail-closed idiom standardised across the ecosystem:

```bash
[[ "${AI_ENABLED:-false}" != "true" ]] && <normal path>
```

Not `== false`. Anything that isn't exactly `true` means off, so a typo can never switch AI
on. (This is the same bug that was fixed in `fallback.sh` — see its `FALLBACK_ENABLED` gate.)

---

## Configuration schema

Follows the existing rule: **thresholds and toggles → `master.conf`; hardware, paths,
container names and per-host identity → `host*.conf`.**

### `host*.conf` — per-host, because only one node actually has the GPU

```bash
# ━━━ Ollama / AI ━━━
    HOST1_OLLAMA_URL="http://localhost:11434"   # empty on nodes without a local Ollama
    HOST1_OLLAMA_CONTAINER="Ollama"             # for docker_watchdog / restart lists
    HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626"
    HOST1_OLLAMA_MODEL="qwen2.5-coder:14b"      # generation
    HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed
```

A node with an empty `OLLAMA_URL` is not an error — it falls through to the resolver and
uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials
it is already waiting on.

### `master.conf` — shared behaviour

```bash
# ━━━ AI ━━━
    AI_ENABLED=false                # master switch — fail-closed, != "true" means off
    AI_CONNECT_TIMEOUT=5            # probe timeout when resolving a host
    AI_REQUEST_TIMEOUT=240          # must clear a cold load — measured 1m45s after tuning.
                                    # KEEP_ALIVE=-1 means this only bites after a restart,
                                    # but a first call that times out is the worst first
                                    # impression a caller can have. Re-measure against a
                                    # real RAG query before fixing this number.
    AI_RESOLVE_CACHE_TTL=300        # don't re-probe the mesh on every script invocation
    AI_MAX_RETRIES=1                # AI is enhancement — do not retry hard

    # Per-feature toggles — enable narration long before enabling decision aid
    AI_ASSIST_REPORTS=false         # tier 1 — digest / coffee report narration
    AI_ASSIST_WATCHDOG=false        # tier 2 — context on a flagged condition
    AI_ASSIST_DISCOVERY=false       # tier 2 — discovery / classification judgment calls
    AI_ASSIST_CLEANUP=false         # tier 2 — HELD orphans, stuck-import triage
    AI_ASSIST_ONBOARD=false         # tier 3 — onboarding / settings assistance

    # Conf writes — separate switch, off by default, see Conf Write Access below
    AI_CONF_WRITE_ENABLED=false
    AI_CONF_WRITE_KEYS=()           # explicit whitelist; never paths or credentials
```

**Per-feature toggles are load-bearing, not decoration.** They are what lets AI narrate the
weekly digest for months before it is ever allowed near a cleanup decision. `AI_ENABLED` is
necessary but not sufficient — every feature stays individually off until it has earned it.

> When these land, `Deployment/master.conf.template` and `Deployment/host.conf.template`
> must be updated in the same pass. That rule is not optional in this repo.

---

## Host resolution

AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node
needs a GPU, a model, or an Ollama container — only the resolver.

`resolve_ollama_host()` mirrors the existing Gitea locator in `git_pull_execute.sh`:

```
Ollama answering on localhost:11434?      → use it            (owner's node)
  else discover_remote_nodes()
       → resolve_tailscale_ip(node)
       → probe each :11434                → first responsive wins
  else                                    → no AI host        (== AI_ENABLED=false)
```

Helpers already exist in `common.sh`: `discover_remote_nodes()` (767),
`resolve_tailscale_ip()` (812), `check_connectivity()` (866).

**Probe the API, not the container.** The Gitea locator checks `docker ps`. Do not copy that
here — a container can be up while the model is unloaded, still pulling, or wedged. Probe
`/api/tags`. Same principle written into `network_watchdog.sh`'s design principles:
*verify the path, not the process*.

**Cache the resolution** in `/tmp` state, like the arr cache. A cleanup script should not pay
a Tailscale round-trip to discover AI it may never call.

### Known consequence

AI lives on HOST1, so **during a fallback — HOST1 down, HOST2 covering — the mesh has no AI.**
That is precisely when a triage assistant would be most useful. Accepted: a second GPU on
HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth
knowing rather than discovering.

---

## Where AI is allowed to act

Ranked by how much damage a wrong answer does.

**Tier 1 — narration and summary (safe, do first)**
- Sunday morning coffee report — turn metrics into prose
- `weekly_health_digest.sh` — summarise, highlight what changed
- Explain *why* a container is crash-looping from its logs

**Tier 2 — triage and context on an existing flag (the real value)**
Places where a script already detects something and stops:
- `system_watchdog` / `stability_watchdog` flags a condition → AI adds context, correlates
  with recent logs, suggests likely cause
- Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work
- `HELD` entries from `arr_download_orphan_cleaner.sh`
- `reverse-anime-leak` from the classification scans — currently report-only *because* it is
  a judgment call. That is exactly the shape AI suits.

**Tier 3 — assisted configuration (needs the guardrails below)**
- Onboarding a new host — the main motivation for conf write access
- AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack

**Never**
- Deciding what to delete
- Choosing a path for any destructive operation
- Anything that bypasses a strike counter, age gate, or verification step

---

## Conf write access

Wanted mainly for onboarding and assisted settings. This is the highest-risk item here.

**Current state: there is no recovery path.**

```
.gitignore:4  Configurations/host*.conf
.gitignore:5  Configurations/master.conf
.gitignore:6  Configurations/*.bak
```

Confs are gitignored — no git history to revert to — and so are the `.bak` files, so the
backup is not versioned either. The only fallback is a single `.bak` slot written by
`conf_upgrade`, and it goes stale immediately:

| file | modified | its `.bak` |
|---|---|---|
| `master.conf` | Jul 28 18:52 | Jul 28 18:52 |
| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** |

A bad write to `master.conf` currently falls back to a file that may predate a month of edits.
**Fix this before any AI writes anything.**

### Required before conf-write ships

1. **Key whitelist, not file access.** Thresholds and toggles only — `*_WARN_GB`,
   `*_STRIKE_LIMIT`, `*_ENABLED`, retention days. Never a path, never a credential, never a
   container list. A wrong threshold is recoverable; a wrong path is what the depth guards
   exist to catch.
2. **Timestamped backups, plural** — `master.conf.2026-08-01T21:00`, retained. Not one
   clobbered slot.
3. **Validate before commit** — `bash -n` the candidate, then confirm `load_config.sh`
   sources it cleanly. Never install a conf that has not been proven to parse.
4. **Diff always logged.** An AI conf change should be at least as visible as a container
   restart.
5. **Lock against concurrent readers** — never rewrite a conf while scripts are mid-run.

**Consider un-ignoring `Configurations/` into a private repo.** Then `git diff` and
`git revert` become the recovery mechanism and the history is free. This overlaps the
existing GitHub-mirror TODO, which is already blocked on the same question.

---

## Security

Ollama has **no authentication of any kind**, and its API includes `DELETE /api/delete`
(wipe models) and `POST /api/pull` (fill the disk). It currently binds `0.0.0.0:11434` with
`OLLAMA_ORIGINS=*` — reachable from the entire LAN, not just Tailscale.

The design only needs loopback (owner) plus the Tailscale interface (mesh). `0.0.0.0` is
strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use
Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than
app-level auth Ollama cannot provide anyway.

**Done 2026-08-01:** `/ext-varaverk` is now mounted `ro` (was `rw` into live prod).
Verified on the running container — `rw=false`.

**Still open:** the LAN exposure above. Deliberately not folded into the tuning rebuild,
since bind-address versus Tailscale ACL is a decision rather than a setting.

---

## Hardware budget

RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby
transcodes on. Do not let AI onto the P2000.

**Tuned and measured 2026-08-01.** These are observed values, not estimates.

| | before | after |
|---|---|---|
| `OLLAMA_NUM_PARALLEL` | 2 | **1** |
| `OLLAMA_KV_CACHE_TYPE` | f16 | **q8_0** |
| `OLLAMA_CONTEXT_LENGTH` | 4096 | **16384** |
| `OLLAMA_FLASH_ATTENTION` | false | **true** |
| VRAM used | 9298 MiB (91%) | **8811 MiB (86%)** |
| warm latency | 2.6s | **1.85s** |
| cold load | 28.6s | **1m45s** |

`/api/ps` confirms `ctx=16384` — the increase is real, not just an env var.

**4× the context for less VRAM than before.** Flash Attention plus the quantized KV cache
more than paid for the increase. Cold load got much slower, which is irrelevant while
`OLLAMA_KEEP_ALIVE=-1` pins both models — but it is felt after any container restart.

### Flash Attention is mandatory, not optional

`OLLAMA_KV_CACHE_TYPE=q8_0` **will not load** without it:

```
llama_init_from_model: V cache quantization requires flash_attn
llama-server process no longer running: exit status 1
```

Quantized K/V cache requires Flash Attention. Supported on Ampere and newer; the 3080
qualifies. If the KV cache type is ever changed back toward a quantized value, Flash
Attention must be on or the model silently fails to load and every call errors.

### Tuning order still matters

If these are ever re-tuned from defaults, the order is load-bearing — raising context first
at high utilisation will OOM:

1. `OLLAMA_NUM_PARALLEL` → 1 (each slot multiplies KV cache)
2. `OLLAMA_FLASH_ATTENTION` → true (prerequisite for the next step)
3. `OLLAMA_KV_CACHE_TYPE` → q8_0 (roughly halves KV memory)
4. *then* `OLLAMA_CONTEXT_LENGTH` upward

32k was considered and rejected — projected ~1000 MiB of KV, leaving under 450 MiB headroom.
16k is the comfortable ceiling for a 14B on this card.

### Concurrency

**You cannot have 14B + long context + real parallelism on 10 GB.** Pick two.
`OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~2s responses,
two or three users serialised is barely noticeable. Multi-user hits are expected to be rare.
So: parallelism stays at 1, the queue absorbs bursts, and the VRAM goes to **context** —
which is what RAG actually needs.

### Applying template changes

**Unraid's "Apply" does not reliably recreate the container.** Observed 2026-08-01: the
template was saved correctly but the container was only *restarted*, so the env vars never
took effect — `Created` stayed unchanged while `Started` advanced. Env changes require a
remove-and-recreate.

Force it with Unraid's own script:

```bash
/usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container Ollama
docker start Ollama      # rebuild stops it — Ollama is not in unraid-autostart
```

Verify with `docker inspect Ollama --format '{{.Created}}'` — the timestamp must move.
Checking the env vars alone is not enough; a restart leaves the old ones in place and
looks like nothing happened.

---

## RAG

No Python on Unraid, and none needed. Everything required is already present:
`sqlite3 3.53`, `jq 1.8`, `node 22`, `php 8.4`, `awk`.

- **chunks + vectors** → SQLite, one table
- **similarity** → cosine in PHP or Node; milliseconds at this corpus size, no vector DB
  container needed
- **embed + generate** → Ollama HTTP, same `curl` pattern as every other integration here

Corpus: **49,654 lines of bash + ~11,970 lines of markdown.** Small.

**Chunking is already solved.** Every script now carries
`PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION /
RUNTIME MODES` at exact, greppable boundaries. Those are semantically coherent units with
stable headings — far better retrieval chunks than fixed-size windows. The `DESIGN PRINCIPLES`
sections encode *why*, which is what the model needs and what code alone never says.

Note `qwen2.5-coder` returns `501 — does not support embeddings`. Embedding is
`nomic-embed-text`'s job. Batch embedding works (n inputs → n vectors in one call) and is
required — indexing 50k lines one HTTP call at a time is not viable.

---

## Scheduled AI

Same orchestrator tiers as everything else, gated on `AI_ENABLED` plus a reachable host.
Natural fits: weekly digest narration, a periodic pass over `HELD`/report-only findings that
have accumulated, post-incident summaries after a watchdog event.

Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal
step like any other, and never blocks the rest of its tier.

---

## UI

- **Dedicated AI page** in the plugin.
- **Persistent conversation across pages.** A floating widget is not required — a chat column
  is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no
  SPA shell, so persistence means conversation state lives server-side keyed by session, with
  the client re-hydrating per page.
- **Per-page AI helpers** — contextual assistance scoped to whatever that page is about.

---

## "Too bad we can't just run the LLM inside Varaverk and cut out Ollama"

There is a real answer: you don't cut Ollama out, you **absorb it**.

Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP
API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers
better than most things manage containers. Ollama becomes just another managed container:

- add to `HOST*_WATCHDOG_CONTAINERS` so `docker_watchdog.sh` keeps it healthy
- add to a restart list so it gets the same proactive treatment as everything else
- give it a fallback tier if AI should survive a host outage
- let `docker_update.sh` handle its image updates

That is more Varaverk-native than embedding a model runtime would be, and it costs nothing
new — the machinery already exists and was audited this session.

---

## Open questions

- Un-ignore `Configurations/` into a private repo for conf history? (blocks conf-write, and
  overlaps the GitHub-mirror TODO)
- Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members?
- Retention/privacy for conversation history — logs may contain paths, container names,
  possibly credentials pasted by a user.
- Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the
  serialisation? Defer until an actual problem is felt.