Add AI entry points, conf schema, and folder docs
ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate, root check, lock, dry-run, status — with Node doing only the vector maths and SQLite blobs, the same split api_cache_writer.sh uses for PHP. AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass. Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes disabled with an empty whitelist. Nothing in the ecosystem consults it.
This commit is contained in:
Gmer4Lfe
+217
@@ -0,0 +1,217 @@
|
||||
# ━━━━━ AI ━━━━━
|
||||
|
||||
Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
|
||||
answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
|
||||
|
||||
Everything here is **off by default and optional**. Varaverk works exactly as well with
|
||||
`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `ai_index.sh` | Build / refresh the retrieval index |
|
||||
| `ai_query.sh` | Ask a question, or search the index directly |
|
||||
| `lib/chunk.js` | Split repo files into retrieval units |
|
||||
| `lib/index.js` | Embed chunks, store vectors in SQLite |
|
||||
| `lib/search.js` | Embed a query, score it, rank results |
|
||||
| `lib/cli.js` | Argument bridge the bash entry points call |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHY THIS WORKS AT ALL ━━━
|
||||
|
||||
**Because the header audit came first.**
|
||||
|
||||
The single worst failure in naive retrieval is a chunk that contains half of one idea and half
|
||||
of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
|
||||
vector. That problem does not exist here, because every script in the repo carries the same six
|
||||
sections at exact, greppable boundaries:
|
||||
|
||||
```
|
||||
PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
|
||||
→ CONFIGURATION → RUNTIME MODES
|
||||
```
|
||||
|
||||
Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
|
||||
heuristics, no tuning.
|
||||
|
||||
**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
|
||||
does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
|
||||
`removeCompletedDownloads` being true on both arrs is intentional. Those live in
|
||||
`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
|
||||
more than an equivalent pile of source.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SECTION ROUTING ━━━
|
||||
|
||||
Every chunk stores its section name as its own column, so a question's *shape* can steer
|
||||
retrieval before similarity is even considered:
|
||||
|
||||
| Question shape | Steered toward |
|
||||
|----------------|----------------|
|
||||
| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
|
||||
| "which variable controls X" | `CONFIGURATION` |
|
||||
| "does this take --dry-run" | `RUNTIME MODES` |
|
||||
| "why is it built this way" | `DESIGN PRINCIPLES` |
|
||||
| "how does X work" | `OPERATIONAL MODEL` |
|
||||
|
||||
Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
|
||||
must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
|
||||
hard filter when you actually want one.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
|
||||
|
||||
Sections alone were not granular enough, and the failure was instructive.
|
||||
|
||||
`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
|
||||
`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
|
||||
correct chunk scored **0.558** — below unrelated chunks from other files — because the other
|
||||
thirteen safeguards dominated the vector.
|
||||
|
||||
The header convention writes each safeguard as a named paragraph: an unindented title, an
|
||||
indented body. Splitting on those titles took the same query to **0.718** and first place.
|
||||
|
||||
```
|
||||
Merge-Run Delete Interlock ← its own chunk
|
||||
--delete is applied only when pass 1 completed...
|
||||
```
|
||||
|
||||
The parent section name is carried onto every sub-chunk, so routing still works. The title
|
||||
detection requires the *next* line to be indented — without that check, any wrapped prose line
|
||||
became a spurious boundary mid-sentence.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT IS INDEXED ━━━
|
||||
|
||||
Roughly 2,900 chunks across ~180 files:
|
||||
|
||||
| Kind | Source |
|
||||
|------|--------|
|
||||
| `header` | Script headers — bash and PHP, sections and named paragraphs |
|
||||
| `readme` | Every `README-*.md` |
|
||||
| `manual` | Every `Manual-*.md` |
|
||||
| `template` | `Deployment/*.template` — the versioned conf schema |
|
||||
| `doc` | Top-level `README.md`, `Manual.md`, design notes |
|
||||
|
||||
**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
|
||||
questions this answers, intent retrieves better and costs far less.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE SAFETY BOUNDARY ━━━
|
||||
|
||||
**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
|
||||
|
||||
`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
|
||||
was never in the repo to begin with. The index therefore describes the full conf schema (via the
|
||||
tracked templates, which carry all the explanatory comments) while structurally **never
|
||||
containing a secret**.
|
||||
|
||||
> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
|
||||
> averaged into a 768-dimension float cannot be found, let alone removed.
|
||||
|
||||
A live conf value the model genuinely needs should arrive through a tool call at query time,
|
||||
subject to redaction — never baked into a vector at index time.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW IT IS BUILT ━━━
|
||||
|
||||
**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
|
||||
per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
|
||||
back up, in exchange for nothing at this scale.
|
||||
|
||||
**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
|
||||
vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
|
||||
reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
|
||||
|
||||
**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
|
||||
no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
|
||||
|
||||
**Node for the maths, bash for everything else.** The bash entry points own configuration,
|
||||
gating, locking and logging exactly as every other Varaverk job does. Node owns only float
|
||||
vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
|
||||
|
||||
> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
|
||||
> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
|
||||
> regenerable in minutes — this is a disposable artefact, not a datastore.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ USAGE ━━━
|
||||
|
||||
```bash
|
||||
# Build (needs AI_ENABLED=true)
|
||||
bash AI/ai_index.sh # incremental
|
||||
bash AI/ai_index.sh --force # full rebuild
|
||||
bash AI/ai_index.sh --dry-run # what would be indexed; contacts nothing
|
||||
bash AI/ai_index.sh --status # size, counts, last build
|
||||
|
||||
# Ask
|
||||
bash AI/ai_query.sh "what stops rsync and the mover running at once"
|
||||
bash AI/ai_query.sh --search "why are the cache writers lockless"
|
||||
bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
|
||||
bash AI/ai_query.sh --json "..." # for other scripts
|
||||
```
|
||||
|
||||
**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
|
||||
When the answer matters, use it — or read the sources the generated answer cites.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
|
||||
|
||||
The generation prompt instructs the model to answer only from retrieved context and to say what
|
||||
is missing rather than fill the gap from general knowledge. That instruction matters here more
|
||||
than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
|
||||
generic answer about rsync, Docker or systemd is worse than no answer.
|
||||
|
||||
It works — and its faithfulness cut both ways on the first real test. Asked which variable
|
||||
controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
|
||||
seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
|
||||
real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
|
||||
|
||||
That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
|
||||
lesson is the operating principle for this whole folder:
|
||||
|
||||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||||
> wrong, check the cited source before blaming the model — it is usually reporting a real
|
||||
> problem in the repo.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS DOES NOT DO ━━━
|
||||
|
||||
- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
|
||||
an empty key whitelist. Nothing in this folder writes a setting.
|
||||
- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
|
||||
per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
|
||||
- **It does not index code.** Headers and docs only.
|
||||
- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
|
||||
- **It is not required.** Every script runs identically with `AI_ENABLED=false`.
|
||||
|
||||
See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
|
||||
Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
|
||||
built before any of that is enabled.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCHEDULING ━━━
|
||||
|
||||
Not scheduled by default. When you want the index to track the repo automatically, the natural
|
||||
home is after a successful pull — the only moment the corpus actually changes:
|
||||
|
||||
```bash
|
||||
# master.conf → DAILY_MAINTENANCE_SCRIPTS, after git_pull_execute.sh
|
||||
"AI/ai_index.sh"
|
||||
```
|
||||
|
||||
`AI_INDEX_ON_PULL` exists in `master.conf` as the intended gate for wiring this into
|
||||
`git_pull_execute.sh` directly. It is not yet consumed by anything — the variable is reserved,
|
||||
not live.
|
||||
|
||||
An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
|
||||
and a pull that changed twelve files costs a few seconds.
|
||||
Reference in New Issue
Block a user