Gmer4Lfe
eb36641523
The index already stored chunk origin and search.js already filtered on it; only the wrapper refused the flag. Intent routing boosts PURPOSE for "what is X", which buried README.md and made the system answer that it had no definition of itself.
233 lines
10 KiB
Markdown
233 lines
10 KiB
Markdown
# ━━━━━ AI ━━━━━
|
||
|
||
Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
|
||
answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
|
||
|
||
Everything here is **off by default and optional**. Varaverk works exactly as well with
|
||
`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
|
||
|
||
| File | Role |
|
||
|------|------|
|
||
| `ai_index.sh` | Build / refresh the retrieval index |
|
||
| `ai_query.sh` | Ask a question, or search the index directly |
|
||
| `lib/chunk.js` | Split repo files into retrieval units |
|
||
| `lib/index.js` | Embed chunks, store vectors in SQLite |
|
||
| `lib/search.js` | Embed a query, score it, rank results |
|
||
| `lib/cli.js` | Argument bridge the bash entry points call |
|
||
|
||
---
|
||
|
||
## ━━━ WHY THIS WORKS AT ALL ━━━
|
||
|
||
**Because the header audit came first.**
|
||
|
||
The single worst failure in naive retrieval is a chunk that contains half of one idea and half
|
||
of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
|
||
vector. That problem does not exist here, because every script in the repo carries the same six
|
||
sections at exact, greppable boundaries:
|
||
|
||
```
|
||
PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
|
||
→ CONFIGURATION → RUNTIME MODES
|
||
```
|
||
|
||
Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
|
||
heuristics, no tuning.
|
||
|
||
**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
|
||
does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
|
||
`removeCompletedDownloads` being true on both arrs is intentional. Those live in
|
||
`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
|
||
more than an equivalent pile of source.
|
||
|
||
---
|
||
|
||
## ━━━ SECTION ROUTING ━━━
|
||
|
||
Every chunk stores its section name as its own column, so a question's *shape* can steer
|
||
retrieval before similarity is even considered:
|
||
|
||
| Question shape | Steered toward |
|
||
|----------------|----------------|
|
||
| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
|
||
| "which variable controls X" | `CONFIGURATION` |
|
||
| "does this take --dry-run" | `RUNTIME MODES` |
|
||
| "why is it built this way" | `DESIGN PRINCIPLES` |
|
||
| "how does X work" | `OPERATIONAL MODEL` |
|
||
|
||
Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
|
||
must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
|
||
hard filter when you actually want one.
|
||
|
||
The boost still has a blind spot worth knowing about: **definitional questions.** "What is
|
||
Varaverk?" matches the `PURPOSE` intent, so every script's one-line PURPOSE gets boosted above
|
||
the top-level prose that actually answers it — and the model correctly replies that the context
|
||
does not define the system. The corpus is fine; `README.md` is indexed. The routing simply
|
||
buries it. `--kind=readme` is the hard filter for that case:
|
||
|
||
```bash
|
||
bash AI/ai_query.sh --kind=readme "what is Varaverk"
|
||
```
|
||
|
||
`--kind` filters on where a chunk came from — `header`, `readme`, `manual`, `template`, `doc` —
|
||
and composes with `--section`. Prefer it over `--section` for "what is" and "why does this
|
||
exist" questions, where the answer is narrative rather than a header field.
|
||
|
||
---
|
||
|
||
## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
|
||
|
||
Sections alone were not granular enough, and the failure was instructive.
|
||
|
||
`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
|
||
`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
|
||
correct chunk scored **0.558** — below unrelated chunks from other files — because the other
|
||
thirteen safeguards dominated the vector.
|
||
|
||
The header convention writes each safeguard as a named paragraph: an unindented title, an
|
||
indented body. Splitting on those titles took the same query to **0.718** and first place.
|
||
|
||
```
|
||
Merge-Run Delete Interlock ← its own chunk
|
||
--delete is applied only when pass 1 completed...
|
||
```
|
||
|
||
The parent section name is carried onto every sub-chunk, so routing still works. The title
|
||
detection requires the *next* line to be indented — without that check, any wrapped prose line
|
||
became a spurious boundary mid-sentence.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT IS INDEXED ━━━
|
||
|
||
Roughly 2,900 chunks across ~180 files:
|
||
|
||
| Kind | Source |
|
||
|------|--------|
|
||
| `header` | Script headers — bash and PHP, sections and named paragraphs |
|
||
| `readme` | Every `README-*.md` |
|
||
| `manual` | Every `Manual-*.md` |
|
||
| `template` | `Deployment/*.template` — the versioned conf schema |
|
||
| `doc` | Top-level `README.md`, `Manual.md`, design notes |
|
||
|
||
**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
|
||
questions this answers, intent retrieves better and costs far less.
|
||
|
||
---
|
||
|
||
## ━━━ THE SAFETY BOUNDARY ━━━
|
||
|
||
**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
|
||
|
||
`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
|
||
was never in the repo to begin with. The index therefore describes the full conf schema (via the
|
||
tracked templates, which carry all the explanatory comments) while structurally **never
|
||
containing a secret**.
|
||
|
||
> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
|
||
> averaged into a 768-dimension float cannot be found, let alone removed.
|
||
|
||
A live conf value the model genuinely needs should arrive through a tool call at query time,
|
||
subject to redaction — never baked into a vector at index time.
|
||
|
||
---
|
||
|
||
## ━━━ HOW IT IS BUILT ━━━
|
||
|
||
**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
|
||
per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
|
||
back up, in exchange for nothing at this scale.
|
||
|
||
**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
|
||
vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
|
||
reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
|
||
|
||
**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
|
||
no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
|
||
|
||
**Node for the maths, bash for everything else.** The bash entry points own configuration,
|
||
gating, locking and logging exactly as every other Varaverk job does. Node owns only float
|
||
vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
|
||
|
||
> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
|
||
> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
|
||
> regenerable in minutes — this is a disposable artefact, not a datastore.
|
||
|
||
---
|
||
|
||
## ━━━ USAGE ━━━
|
||
|
||
```bash
|
||
# Build (needs AI_ENABLED=true)
|
||
bash AI/ai_index.sh # incremental
|
||
bash AI/ai_index.sh --force # full rebuild
|
||
bash AI/ai_index.sh --dry-run # what would be indexed; contacts nothing
|
||
bash AI/ai_index.sh --status # size, counts, last build
|
||
|
||
# Ask
|
||
bash AI/ai_query.sh "what stops rsync and the mover running at once"
|
||
bash AI/ai_query.sh --search "why are the cache writers lockless"
|
||
bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
|
||
bash AI/ai_query.sh --kind=readme "what is Varaverk" # definitional / narrative
|
||
bash AI/ai_query.sh --json "..." # for other scripts
|
||
```
|
||
|
||
**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
|
||
When the answer matters, use it — or read the sources the generated answer cites.
|
||
|
||
---
|
||
|
||
## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
|
||
|
||
The generation prompt instructs the model to answer only from retrieved context and to say what
|
||
is missing rather than fill the gap from general knowledge. That instruction matters here more
|
||
than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
|
||
generic answer about rsync, Docker or systemd is worse than no answer.
|
||
|
||
It works — and its faithfulness cut both ways on the first real test. Asked which variable
|
||
controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
|
||
seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
|
||
real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
|
||
|
||
That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
|
||
lesson is the operating principle for this whole folder:
|
||
|
||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||
> wrong, check the cited source before blaming the model — it is usually reporting a real
|
||
> problem in the repo.
|
||
|
||
---
|
||
|
||
## ━━━ WHAT THIS DOES NOT DO ━━━
|
||
|
||
- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
|
||
an empty key whitelist. Nothing in this folder writes a setting.
|
||
- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
|
||
per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
|
||
- **It does not index code.** Headers and docs only.
|
||
- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
|
||
- **It is not required.** Every script runs identically with `AI_ENABLED=false`.
|
||
|
||
See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
|
||
Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
|
||
built before any of that is enabled.
|
||
|
||
---
|
||
|
||
## ━━━ SCHEDULING ━━━
|
||
|
||
Not scheduled by default. When you want the index to track the repo automatically, the natural
|
||
home is after a successful pull — the only moment the corpus actually changes:
|
||
|
||
```bash
|
||
# master.conf → DAILY_MAINTENANCE_SCRIPTS, after git_pull_execute.sh
|
||
"AI/ai_index.sh"
|
||
```
|
||
|
||
`AI_INDEX_ON_PULL` exists in `master.conf` as the intended gate for wiring this into
|
||
`git_pull_execute.sh` directly. It is not yet consumed by anything — the variable is reserved,
|
||
not live.
|
||
|
||
An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
|
||
and a pull that changed twelve files costs a few seconds.
|