# ━━━━━ AI ━━━━━

Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.

Everything here is **off by default and optional**. Varaverk works exactly as well with
`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.

| File | Role |
|------|------|
| `ai_index.sh` | Build / refresh the retrieval index |
| `ai_query.sh` | Ask a question, or search the index directly |
| `lib/chunk.js` | Split repo files into retrieval units |
| `lib/index.js` | Embed chunks, store vectors in SQLite |
| `lib/search.js` | Embed a query, score it, rank results |
| `lib/cli.js` | Argument bridge the bash entry points call |

---

## ━━━ WHY THIS WORKS AT ALL ━━━

**Because the header audit came first.**

The single worst failure in naive retrieval is a chunk that contains half of one idea and half
of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
vector. That problem does not exist here, because every script in the repo carries the same six
sections at exact, greppable boundaries:

```
PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
        → CONFIGURATION → RUNTIME MODES
```

Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
heuristics, no tuning.

**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
`removeCompletedDownloads` being true on both arrs is intentional. Those live in
`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
more than an equivalent pile of source.

---

## ━━━ SECTION ROUTING ━━━

Every chunk stores its section name as its own column, so a question's *shape* can steer
retrieval before similarity is even considered:

| Question shape | Steered toward |
|----------------|----------------|
| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
| "which variable controls X" | `CONFIGURATION` |
| "does this take --dry-run" | `RUNTIME MODES` |
| "why is it built this way" | `DESIGN PRINCIPLES` |
| "how does X work" | `OPERATIONAL MODEL` |

Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
hard filter when you actually want one.

The boost still has a blind spot worth knowing about: **definitional questions.** "What is
Varaverk?" matches the `PURPOSE` intent, so every script's one-line PURPOSE gets boosted above
the top-level prose that actually answers it — and the model correctly replies that the context
does not define the system. The corpus is fine; `README.md` is indexed. The routing simply
buries it. `--kind=readme` is the hard filter for that case:

```bash
bash AI/ai_query.sh --kind=readme "what is Varaverk"
```

`--kind` filters on where a chunk came from — `header`, `readme`, `manual`, `template`, `doc`, `ui` —
and composes with `--section`. Prefer it over `--section` for "what is" and "why does this
exist" questions, where the answer is narrative rather than a header field.

---

## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━

Sections alone were not granular enough, and the failure was instructive.

`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
correct chunk scored **0.558** — below unrelated chunks from other files — because the other
thirteen safeguards dominated the vector.

The header convention writes each safeguard as a named paragraph: an unindented title, an
indented body. Splitting on those titles took the same query to **0.718** and first place.

```
Merge-Run Delete Interlock            ← its own chunk
  --delete is applied only when pass 1 completed...
```

The parent section name is carried onto every sub-chunk, so routing still works. The title
detection requires the *next* line to be indented — without that check, any wrapped prose line
became a spurious boundary mid-sentence.

---

## ━━━ WHAT IS INDEXED ━━━

Roughly 2,900 chunks across ~180 files:

| Kind | Source |
|------|--------|
| `header` | Script headers — bash and PHP, sections and named paragraphs |
| `readme` | Every `README-*.md` |
| `manual` | Every `Manual-*.md` |
| `template` | `Deployment/*.template` — the versioned conf schema |
| `doc` | Top-level `README.md`, `Manual.md`, design notes |
| `ui` | `Plugin/unraid/pages/readme/*.md` — the WebGUI's own help panels |

**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
questions this answers, intent retrieves better and costs far less.

---

## ━━━ THE SAFETY BOUNDARY ━━━

**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.

`Configurations/` and `data/` are gitignored, so every file holding a credential
was never in the repo to begin with. The index therefore describes the full conf schema (via the
tracked templates, which carry all the explanatory comments) while structurally **never
containing a secret**.

> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
> averaged into a 768-dimension float cannot be found, let alone removed.

A live conf value the model genuinely needs should arrive through a tool call at query time,
subject to redaction — never baked into a vector at index time.

---

## ━━━ HOW IT IS BUILT ━━━

**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
back up, in exchange for nothing at this scale.

**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
reads the same blobs with `unpack('f*', $blob)` when the UI needs them.

**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
no-op refresh takes about 70 ms; a full rebuild takes a few minutes.

**Node for the maths, bash for everything else.** The bash entry points own configuration,
gating, locking and logging exactly as every other Varaverk job does. Node owns only float
vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.

> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
> regenerable in minutes — this is a disposable artefact, not a datastore.

---

## ━━━ USAGE ━━━

```bash
# Build (needs AI_ENABLED=true)
bash AI/ai_index.sh                  # incremental
bash AI/ai_index.sh --force          # full rebuild
bash AI/ai_index.sh --dry-run        # what would be indexed; contacts nothing
bash AI/ai_index.sh --status         # size, counts, last build

# Ask
bash AI/ai_query.sh "what stops rsync and the mover running at once"
bash AI/ai_query.sh --search "why are the cache writers lockless"
bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
bash AI/ai_query.sh --kind=readme "what is Varaverk"          # definitional / narrative
bash AI/ai_query.sh --json "..."     # for other scripts
```

**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
When the answer matters, use it — or read the sources the generated answer cites.

---

## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━

The generation prompt instructs the model to answer only from retrieved context and to say what
is missing rather than fill the gap from general knowledge. That instruction matters here more
than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
generic answer about rsync, Docker or systemd is worse than no answer.

It works — and its faithfulness cut both ways on the first real test. Asked which variable
controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
real value is 300 — and the model was quoting the header verbatim. **The header was stale.**

That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
lesson is the operating principle for this whole folder:

> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
> wrong, check the cited source before blaming the model — it is usually reporting a real
> problem in the repo.

---

## ━━━ WHAT THIS DOES NOT DO ━━━

- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
  an empty key whitelist. Nothing in this folder writes a setting.
- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
  per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
- **It does not index code.** Headers and docs only.
- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
- **It is not required.** Every script runs identically with `AI_ENABLED=false`.

See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
built before any of that is enabled.

---

## ━━━ SCHEDULING ━━━

The index tracks the repo automatically, from the only moment the corpus actually changes — a
successful pull. `git_pull_execute.sh` re-indexes behind three gates: the pull succeeded and
changed tracked files, `AI_INDEX_ON_PULL=true`, and `AI_ENABLED=true`. It is never fatal — a
failed index leaves the previous one in place and the pull still reports success.

No cron entry and no `DAILY_MAINTENANCE_SCRIPTS` line are needed; the daily pull carries it.

An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
and a pull that changed twelve files costs a few seconds.

---

## ━━━ PROFILES ━━━

A profile is a contract plus a set of inputs. `Plugin/unraid/include/ai_profiles.php` is the one
definition of both, read by the endpoint, the worker, the shared chat include and the Scheduler
dock.

| Profile | Turns | Retrieves | Notes |
|---|---|---|---|
| `varaverk` | 3 | yes | answers only from the index, with citations. The default. |
| `chat` | 8 | **no** | ordinary conversation. Holds zero capabilities, deliberately. |
| `code` | 4 | no | drafts shell for Custom Scripts; scans its own output for destructive ops |
| `troubleshoot` | 2 | yes | reasons from evidence — an open log, or one you name. May file bug reports. |

Capabilities are granted per profile — retrieval, live health, run evidence, scoped log,
incidents, conf lookup, bug filing, code scanning. `chat` holding an empty list is a guarantee,
not an oversight: anything added to it stops being general chat and becomes an assistant that
sometimes invents claims about this installation.

### Routing out of General Chat

`chat` hands a question to whichever profile fits, decided by `vv_ai_route_from_chat()`. Ordered
most specific first, because these overlap on purpose:

| Question | Goes to | Why |
|---|---|---|
| "write me a script that prunes logs" | `code` | asked for something written |
| "why did the daily orch fail" | `troubleshoot` | diagnostic phrasing **and** something here to diagnose |
| "how did the daily orch go" | `varaverk` | about this install, but not a fault |
| "what does arr_sync.sh do" | `varaverk` | names a script, wants documentation |
| "why is the sky blue" | stays `chat` | diagnostic phrasing about nothing here |

`code` is checked first because it is the only intent about a thing that does not exist yet, so
nothing else can claim it — and it is anchored on the verb, which is what keeps "write me a
script" apart from "what does this script do". Escalation adds capability, so a wrong escalation
costs more than a missed one: anything unrecognised stays in `chat`, the profile that cannot
invent claims about this system. The worker reverts to `chat` anyway if retrieval comes back
empty.

The answer opens with one line naming the profile that took it, because the button still shows
the one you picked and an answer arriving under a different contract otherwise reads as the
assistant ignoring you.

Routing is asserted by `Plugin/unraid/Tools/ai_explain_check.sh` against
`ai_explain_fixtures.txt` — every case runs through the worker's `--explain` mode, which stops
where deterministic assembly ends and never calls the model.

This used to live in five places — history depth in the endpoint, capabilities in `include/ai.php`,
label and depth again in JavaScript, a prompt branch in the worker, and a label map on the
Scheduler page. They had already drifted: the JavaScript knew three profiles where PHP knew four.
The system prompts still live in `Tools/ai_chat_worker.php`, because they have one reader and
moving them would relocate the most delicate text in the subsystem without removing a duplicate.

## ━━━ CONVERSATIONS ━━━

Chats are stored server-side under `AI_DATA_DIR/ai_chats/`, one JSON file each, saved
automatically when a turn completes and pruned to `AI_CHAT_HISTORY_MAX` (default 10, oldest
first by creation).

There is no Save button. A conversation worth keeping is not reliably one you knew was worth
keeping while you were having it.

The same store backs the AI tab and the Monitor tab's AI row, so a thread started on the
dashboard is the one you carry on in the tab. Messages are re-validated per message on the way
in — a stored chat is replayed into a later prompt when reopened, so an unchecked role written
there would be an injection that survives a reload rather than one turn.

Reopened chats render as plain turns: sources, reasoning and timings describe one generation and
are not stored, because redrawing them beside a transcript that may be continued under a
different profile would be citing evidence for an answer no longer being made.

### Secrets are redacted on the way to disk

A conversation about settings is a conversation containing credentials — asking for an API key to
be changed means typing one. Message bodies are redacted in `vv_ai_chat_save()`, and the question
is redacted again before it reaches `ai.log`.

**On the way out, never in flight.** The live turn keeps the real value, because the model needs
it to carry out what was asked. What it does not need is that value still in the transcript a week
later — and a stored chat is replayed into a later prompt when reopened, so an unredacted one
would hand the credential back on every subsequent turn, indefinitely.

Two passes, because they catch different things:

| Pass | Catches | Method |
|---|---|---|
| Known values | a credential this host already holds | exact match against secret-shaped conf keys, longest first |
| Assignment shapes | a credential arriving that is not in the conf yet | `NAME=value`, `"api_key": value`, "set the token to …" |

The second pass is the one that matters for settings changes: *"change the Emby API key to X"* is
a secret arriving, and X matches nothing on disk until after the write it is requesting.

Ordinary prose is left alone — the patterns anchor on a secret-shaped *name*, so `CACHE_WARN_GB=100`
and "turn off the zfs scrub" pass through untouched. `vv_conf_key_is_secret()` is shared with the
conf audit log, so the two cannot disagree about what counts as a secret.

## ━━━ TOKEN ACCOUNTING ━━━

Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):

```
date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s
2026-08-04|22:03:51|host1|varaverk|cli|2041|318|61.4
```

Both paths write it — this CLI (`source=cli`) and the WebGUI worker (`source=webgui`) — so the
totals are not quietly the tab's alone. `ai_query.sh` passes `--token-db` and `--token-host`;
called by hand without them, `cli.js` simply skips the row rather than guessing a path, because
this file never reads conf itself.

Read it on the plugin's AI tab, which aggregates today / last 7 days / all time, per host. Or
straight from the shell, since it is just a delimited file:

```bash
# tokens used today
awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai/ai_token_history.db
```

**The host column is where the turn ran, not where the file is read.** Each host writes only its
own rows.

`ai_token_sync.sh` pulls each partner's ledger into `$AI_TOKEN_CACHE_DIR/<slot>.tokens.db` (`/tmp/varaverk/ai/`) — the
same trick `conf_sync.sh` uses for partner confs, and it runs from
`INTERMEDIATE_MAINTENANCE_SCRIPTS` every four hours. The tab then reads every ledger it can see,
so a fleet total is a fleet total.

Pull only, no push: nothing here is needed by anyone else, and a reader that fetches its own data
controls its own freshness instead of depending on the partner's cron. A partner file may only
contribute rows whose `host` column matches its filename — a ledger copied into the wrong slot
would otherwise be double-counted against a total that still looked plausible.

The cache is tmpfs with **no save/restore pair**, unlike the conf cache. Stale counters are worse
than absent ones: absent renders as "not collected here", stale renders as fact. An unreachable
partner leaves its file alone and logs at info, because a partner being down for weeks is a
normal state, not an incident.

Pruning is by row count (`AI_TOKEN_RETAIN_ROWS`, default 20000) and happens on write, but only
once the file passes a size threshold — an ordinary turn costs a `stat()` and an append. The CLI
deliberately does not prune: duplicating a read-modify-write of the whole file in a second
language is how the two drift apart.
