320 lines
15 KiB
Markdown
320 lines
15 KiB
Markdown
# ━━━━━ AI ━━━━━
|
||
|
||
Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
|
||
answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
|
||
|
||
Everything here is **off by default and optional**. Varaverk works exactly as well with
|
||
`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
|
||
|
||
| File | Role |
|
||
|------|------|
|
||
| `ai_index.sh` | Build / refresh the retrieval index |
|
||
| `ai_query.sh` | Ask a question, or search the index directly |
|
||
| `lib/chunk.js` | Split repo files into retrieval units |
|
||
| `lib/index.js` | Embed chunks, store vectors in SQLite |
|
||
| `lib/search.js` | Embed a query, score it, rank results |
|
||
| `lib/cli.js` | Argument bridge the bash entry points call |
|
||
|
||
---
|
||
|
||
## ━━━ WHY THIS WORKS AT ALL ━━━
|
||
|
||
**Because the header audit came first.**
|
||
|
||
The single worst failure in naive retrieval is a chunk that contains half of one idea and half
|
||
of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
|
||
vector. That problem does not exist here, because every script in the repo carries the same six
|
||
sections at exact, greppable boundaries:
|
||
|
||
```
|
||
PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
|
||
→ CONFIGURATION → RUNTIME MODES
|
||
```
|
||
|
||
Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
|
||
heuristics, no tuning.
|
||
|
||
**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
|
||
does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
|
||
`removeCompletedDownloads` being true on both arrs is intentional. Those live in
|
||
`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
|
||
more than an equivalent pile of source.
|
||
|
||
---
|
||
|
||
## ━━━ SECTION ROUTING ━━━
|
||
|
||
Every chunk stores its section name as its own column, so a question's *shape* can steer
|
||
retrieval before similarity is even considered:
|
||
|
||
| Question shape | Steered toward |
|
||
|----------------|----------------|
|
||
| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
|
||
| "which variable controls X" | `CONFIGURATION` |
|
||
| "does this take --dry-run" | `RUNTIME MODES` |
|
||
| "why is it built this way" | `DESIGN PRINCIPLES` |
|
||
| "how does X work" | `OPERATIONAL MODEL` |
|
||
|
||
Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
|
||
must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
|
||
hard filter when you actually want one.
|
||
|
||
The boost still has a blind spot worth knowing about: **definitional questions.** "What is
|
||
Varaverk?" matches the `PURPOSE` intent, so every script's one-line PURPOSE gets boosted above
|
||
the top-level prose that actually answers it — and the model correctly replies that the context
|
||
does not define the system. The corpus is fine; `README.md` is indexed. The routing simply
|
||
buries it. `--kind=readme` is the hard filter for that case:
|
||
|
||
```bash
|
||
bash AI/ai_query.sh --kind=readme "what is Varaverk"
|
||
```
|
||
|
||
`--kind` filters on where a chunk came from — `header`, `readme`, `manual`, `template`, `doc` —
|
||
and composes with `--section`. Prefer it over `--section` for "what is" and "why does this
|
||
exist" questions, where the answer is narrative rather than a header field.
|
||
|
||
---
|
||
|
||
## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
|
||
|
||
Sections alone were not granular enough, and the failure was instructive.
|
||
|
||
`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
|
||
`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
|
||
correct chunk scored **0.558** — below unrelated chunks from other files — because the other
|
||
thirteen safeguards dominated the vector.
|
||
|
||
The header convention writes each safeguard as a named paragraph: an unindented title, an
|
||
indented body. Splitting on those titles took the same query to **0.718** and first place.
|
||
|
||
```
|
||
Merge-Run Delete Interlock ← its own chunk
|
||
--delete is applied only when pass 1 completed...
|
||
```
|
||
|
||
The parent section name is carried onto every sub-chunk, so routing still works. The title
|
||
detection requires the *next* line to be indented — without that check, any wrapped prose line
|
||
became a spurious boundary mid-sentence.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT IS INDEXED ━━━
|
||
|
||
Roughly 2,900 chunks across ~180 files:
|
||
|
||
| Kind | Source |
|
||
|------|--------|
|
||
| `header` | Script headers — bash and PHP, sections and named paragraphs |
|
||
| `readme` | Every `README-*.md` |
|
||
| `manual` | Every `Manual-*.md` |
|
||
| `template` | `Deployment/*.template` — the versioned conf schema |
|
||
| `doc` | Top-level `README.md`, `Manual.md`, design notes |
|
||
|
||
**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
|
||
questions this answers, intent retrieves better and costs far less.
|
||
|
||
---
|
||
|
||
## ━━━ THE SAFETY BOUNDARY ━━━
|
||
|
||
**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
|
||
|
||
`Configurations/` and `data/` are gitignored, so every file holding a credential
|
||
was never in the repo to begin with. The index therefore describes the full conf schema (via the
|
||
tracked templates, which carry all the explanatory comments) while structurally **never
|
||
containing a secret**.
|
||
|
||
> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
|
||
> averaged into a 768-dimension float cannot be found, let alone removed.
|
||
|
||
A live conf value the model genuinely needs should arrive through a tool call at query time,
|
||
subject to redaction — never baked into a vector at index time.
|
||
|
||
---
|
||
|
||
## ━━━ HOW IT IS BUILT ━━━
|
||
|
||
**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
|
||
per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
|
||
back up, in exchange for nothing at this scale.
|
||
|
||
**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
|
||
vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
|
||
reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
|
||
|
||
**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
|
||
no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
|
||
|
||
**Node for the maths, bash for everything else.** The bash entry points own configuration,
|
||
gating, locking and logging exactly as every other Varaverk job does. Node owns only float
|
||
vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
|
||
|
||
> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
|
||
> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
|
||
> regenerable in minutes — this is a disposable artefact, not a datastore.
|
||
|
||
---
|
||
|
||
## ━━━ USAGE ━━━
|
||
|
||
```bash
|
||
# Build (needs AI_ENABLED=true)
|
||
bash AI/ai_index.sh # incremental
|
||
bash AI/ai_index.sh --force # full rebuild
|
||
bash AI/ai_index.sh --dry-run # what would be indexed; contacts nothing
|
||
bash AI/ai_index.sh --status # size, counts, last build
|
||
|
||
# Ask
|
||
bash AI/ai_query.sh "what stops rsync and the mover running at once"
|
||
bash AI/ai_query.sh --search "why are the cache writers lockless"
|
||
bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
|
||
bash AI/ai_query.sh --kind=readme "what is Varaverk" # definitional / narrative
|
||
bash AI/ai_query.sh --json "..." # for other scripts
|
||
```
|
||
|
||
**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
|
||
When the answer matters, use it — or read the sources the generated answer cites.
|
||
|
||
---
|
||
|
||
## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
|
||
|
||
The generation prompt instructs the model to answer only from retrieved context and to say what
|
||
is missing rather than fill the gap from general knowledge. That instruction matters here more
|
||
than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
|
||
generic answer about rsync, Docker or systemd is worse than no answer.
|
||
|
||
It works — and its faithfulness cut both ways on the first real test. Asked which variable
|
||
controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
|
||
seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
|
||
real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
|
||
|
||
That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
|
||
lesson is the operating principle for this whole folder:
|
||
|
||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||
> wrong, check the cited source before blaming the model — it is usually reporting a real
|
||
> problem in the repo.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT THIS DOES NOT DO ━━━
|
||
|
||
- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
|
||
an empty key whitelist. Nothing in this folder writes a setting.
|
||
- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
|
||
per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
|
||
- **It does not index code.** Headers and docs only.
|
||
- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
|
||
- **It is not required.** Every script runs identically with `AI_ENABLED=false`.
|
||
|
||
See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
|
||
Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
|
||
built before any of that is enabled.
|
||
|
||
---
|
||
|
||
## ━━━ SCHEDULING ━━━
|
||
|
||
The index tracks the repo automatically, from the only moment the corpus actually changes — a
|
||
successful pull. `git_pull_execute.sh` re-indexes behind three gates: the pull succeeded and
|
||
changed tracked files, `AI_INDEX_ON_PULL=true`, and `AI_ENABLED=true`. It is never fatal — a
|
||
failed index leaves the previous one in place and the pull still reports success.
|
||
|
||
No cron entry and no `DAILY_MAINTENANCE_SCRIPTS` line are needed; the daily pull carries it.
|
||
|
||
An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
|
||
and a pull that changed twelve files costs a few seconds.
|
||
|
||
---
|
||
|
||
## ━━━ PROFILES ━━━
|
||
|
||
A profile is a contract plus a set of inputs. `Plugin/unraid/include/ai_profiles.php` is the one
|
||
definition of both, read by the endpoint, the worker, the shared chat include and the Scheduler
|
||
dock.
|
||
|
||
| Profile | Turns | Retrieves | Notes |
|
||
|---|---|---|---|
|
||
| `varaverk` | 3 | yes | answers only from the index, with citations. The default. |
|
||
| `chat` | 8 | **no** | ordinary conversation. Holds zero capabilities, deliberately. |
|
||
| `code` | 4 | no | drafts shell for Custom Scripts; scans its own output for destructive ops |
|
||
| `troubleshoot` | 2 | yes | reasons from an open log first, docs second. May file bug reports. |
|
||
|
||
Capabilities are granted per profile — retrieval, live health, run evidence, scoped log,
|
||
incidents, conf lookup, bug filing, code scanning. `chat` holding an empty list is a guarantee,
|
||
not an oversight: anything added to it stops being general chat and becomes an assistant that
|
||
sometimes invents claims about this installation.
|
||
|
||
`chat` escalates to `varaverk` on its own when a question is genuinely about Varaverk, and
|
||
reverts if the index turns out to have nothing — so the loose profile is safe to sit in.
|
||
|
||
This used to live in five places — history depth in the endpoint, capabilities in `include/ai.php`,
|
||
label and depth again in JavaScript, a prompt branch in the worker, and a label map on the
|
||
Scheduler page. They had already drifted: the JavaScript knew three profiles where PHP knew four.
|
||
The system prompts still live in `Tools/ai_chat_worker.php`, because they have one reader and
|
||
moving them would relocate the most delicate text in the subsystem without removing a duplicate.
|
||
|
||
## ━━━ CONVERSATIONS ━━━
|
||
|
||
Chats are stored server-side under `AI_DATA_DIR/ai_chats/`, one JSON file each, saved
|
||
automatically when a turn completes and pruned to `AI_CHAT_HISTORY_MAX` (default 10, oldest
|
||
first by creation).
|
||
|
||
There is no Save button. A conversation worth keeping is not reliably one you knew was worth
|
||
keeping while you were having it.
|
||
|
||
The same store backs the AI tab and the Monitor tab's AI row, so a thread started on the
|
||
dashboard is the one you carry on in the tab. Messages are re-validated per message on the way
|
||
in — a stored chat is replayed into a later prompt when reopened, so an unchecked role written
|
||
there would be an injection that survives a reload rather than one turn.
|
||
|
||
Reopened chats render as plain turns: sources, reasoning and timings describe one generation and
|
||
are not stored, because redrawing them beside a transcript that may be continued under a
|
||
different profile would be citing evidence for an answer no longer being made.
|
||
|
||
## ━━━ TOKEN ACCOUNTING ━━━
|
||
|
||
Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
|
||
|
||
```
|
||
date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s
|
||
2026-08-04|22:03:51|host1|varaverk|cli|2041|318|61.4
|
||
```
|
||
|
||
Both paths write it — this CLI (`source=cli`) and the WebGUI worker (`source=webgui`) — so the
|
||
totals are not quietly the tab's alone. `ai_query.sh` passes `--token-db` and `--token-host`;
|
||
called by hand without them, `cli.js` simply skips the row rather than guessing a path, because
|
||
this file never reads conf itself.
|
||
|
||
Read it on the plugin's AI tab, which aggregates today / last 7 days / all time, per host. Or
|
||
straight from the shell, since it is just a delimited file:
|
||
|
||
```bash
|
||
# tokens used today
|
||
awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai/ai_token_history.db
|
||
```
|
||
|
||
**The host column is where the turn ran, not where the file is read.** Each host writes only its
|
||
own rows.
|
||
|
||
`ai_token_sync.sh` pulls each partner's ledger into `$AI_TOKEN_CACHE_DIR/<slot>.tokens.db` (`/tmp/varaverk/ai/`) — the
|
||
same trick `conf_sync.sh` uses for partner confs, and it runs from
|
||
`INTERMEDIATE_MAINTENANCE_SCRIPTS` every four hours. The tab then reads every ledger it can see,
|
||
so a fleet total is a fleet total.
|
||
|
||
Pull only, no push: nothing here is needed by anyone else, and a reader that fetches its own data
|
||
controls its own freshness instead of depending on the partner's cron. A partner file may only
|
||
contribute rows whose `host` column matches its filename — a ledger copied into the wrong slot
|
||
would otherwise be double-counted against a total that still looked plausible.
|
||
|
||
The cache is tmpfs with **no save/restore pair**, unlike the conf cache. Stale counters are worse
|
||
than absent ones: absent renders as "not collected here", stale renders as fact. An unreachable
|
||
partner leaves its file alone and logs at info, because a partner being down for weeks is a
|
||
normal state, not an incident.
|
||
|
||
Pruning is by row count (`AI_TOKEN_RETAIN_ROWS`, default 20000) and happens on write, but only
|
||
once the file passes a size threshold — an ordinary turn costs a `stat()` and an append. The CLI
|
||
deliberately does not prune: duplicating a read-modify-write of the whole file in a second
|
||
language is how the two drift apart.
|