Add AI entry points, conf schema, and folder docs
ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate, root check, lock, dry-run, status — with Node doing only the vector maths and SQLite blobs, the same split api_cache_writer.sh uses for PHP. AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass. Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes disabled with an empty whitelist. Nothing in the ecosystem consults it.
This commit is contained in:
Gmer4Lfe
+217
@@ -0,0 +1,217 @@
|
||||
# ━━━━━ AI ━━━━━
|
||||
|
||||
Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
|
||||
answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
|
||||
|
||||
Everything here is **off by default and optional**. Varaverk works exactly as well with
|
||||
`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `ai_index.sh` | Build / refresh the retrieval index |
|
||||
| `ai_query.sh` | Ask a question, or search the index directly |
|
||||
| `lib/chunk.js` | Split repo files into retrieval units |
|
||||
| `lib/index.js` | Embed chunks, store vectors in SQLite |
|
||||
| `lib/search.js` | Embed a query, score it, rank results |
|
||||
| `lib/cli.js` | Argument bridge the bash entry points call |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHY THIS WORKS AT ALL ━━━
|
||||
|
||||
**Because the header audit came first.**
|
||||
|
||||
The single worst failure in naive retrieval is a chunk that contains half of one idea and half
|
||||
of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
|
||||
vector. That problem does not exist here, because every script in the repo carries the same six
|
||||
sections at exact, greppable boundaries:
|
||||
|
||||
```
|
||||
PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
|
||||
→ CONFIGURATION → RUNTIME MODES
|
||||
```
|
||||
|
||||
Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
|
||||
heuristics, no tuning.
|
||||
|
||||
**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
|
||||
does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
|
||||
`removeCompletedDownloads` being true on both arrs is intentional. Those live in
|
||||
`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
|
||||
more than an equivalent pile of source.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SECTION ROUTING ━━━
|
||||
|
||||
Every chunk stores its section name as its own column, so a question's *shape* can steer
|
||||
retrieval before similarity is even considered:
|
||||
|
||||
| Question shape | Steered toward |
|
||||
|----------------|----------------|
|
||||
| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
|
||||
| "which variable controls X" | `CONFIGURATION` |
|
||||
| "does this take --dry-run" | `RUNTIME MODES` |
|
||||
| "why is it built this way" | `DESIGN PRINCIPLES` |
|
||||
| "how does X work" | `OPERATIONAL MODEL` |
|
||||
|
||||
Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
|
||||
must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
|
||||
hard filter when you actually want one.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
|
||||
|
||||
Sections alone were not granular enough, and the failure was instructive.
|
||||
|
||||
`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
|
||||
`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
|
||||
correct chunk scored **0.558** — below unrelated chunks from other files — because the other
|
||||
thirteen safeguards dominated the vector.
|
||||
|
||||
The header convention writes each safeguard as a named paragraph: an unindented title, an
|
||||
indented body. Splitting on those titles took the same query to **0.718** and first place.
|
||||
|
||||
```
|
||||
Merge-Run Delete Interlock ← its own chunk
|
||||
--delete is applied only when pass 1 completed...
|
||||
```
|
||||
|
||||
The parent section name is carried onto every sub-chunk, so routing still works. The title
|
||||
detection requires the *next* line to be indented — without that check, any wrapped prose line
|
||||
became a spurious boundary mid-sentence.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT IS INDEXED ━━━
|
||||
|
||||
Roughly 2,900 chunks across ~180 files:
|
||||
|
||||
| Kind | Source |
|
||||
|------|--------|
|
||||
| `header` | Script headers — bash and PHP, sections and named paragraphs |
|
||||
| `readme` | Every `README-*.md` |
|
||||
| `manual` | Every `Manual-*.md` |
|
||||
| `template` | `Deployment/*.template` — the versioned conf schema |
|
||||
| `doc` | Top-level `README.md`, `Manual.md`, design notes |
|
||||
|
||||
**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
|
||||
questions this answers, intent retrieves better and costs far less.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE SAFETY BOUNDARY ━━━
|
||||
|
||||
**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
|
||||
|
||||
`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
|
||||
was never in the repo to begin with. The index therefore describes the full conf schema (via the
|
||||
tracked templates, which carry all the explanatory comments) while structurally **never
|
||||
containing a secret**.
|
||||
|
||||
> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
|
||||
> averaged into a 768-dimension float cannot be found, let alone removed.
|
||||
|
||||
A live conf value the model genuinely needs should arrive through a tool call at query time,
|
||||
subject to redaction — never baked into a vector at index time.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW IT IS BUILT ━━━
|
||||
|
||||
**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
|
||||
per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
|
||||
back up, in exchange for nothing at this scale.
|
||||
|
||||
**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
|
||||
vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
|
||||
reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
|
||||
|
||||
**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
|
||||
no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
|
||||
|
||||
**Node for the maths, bash for everything else.** The bash entry points own configuration,
|
||||
gating, locking and logging exactly as every other Varaverk job does. Node owns only float
|
||||
vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
|
||||
|
||||
> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
|
||||
> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
|
||||
> regenerable in minutes — this is a disposable artefact, not a datastore.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ USAGE ━━━
|
||||
|
||||
```bash
|
||||
# Build (needs AI_ENABLED=true)
|
||||
bash AI/ai_index.sh # incremental
|
||||
bash AI/ai_index.sh --force # full rebuild
|
||||
bash AI/ai_index.sh --dry-run # what would be indexed; contacts nothing
|
||||
bash AI/ai_index.sh --status # size, counts, last build
|
||||
|
||||
# Ask
|
||||
bash AI/ai_query.sh "what stops rsync and the mover running at once"
|
||||
bash AI/ai_query.sh --search "why are the cache writers lockless"
|
||||
bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
|
||||
bash AI/ai_query.sh --json "..." # for other scripts
|
||||
```
|
||||
|
||||
**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
|
||||
When the answer matters, use it — or read the sources the generated answer cites.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
|
||||
|
||||
The generation prompt instructs the model to answer only from retrieved context and to say what
|
||||
is missing rather than fill the gap from general knowledge. That instruction matters here more
|
||||
than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
|
||||
generic answer about rsync, Docker or systemd is worse than no answer.
|
||||
|
||||
It works — and its faithfulness cut both ways on the first real test. Asked which variable
|
||||
controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
|
||||
seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
|
||||
real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
|
||||
|
||||
That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
|
||||
lesson is the operating principle for this whole folder:
|
||||
|
||||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||||
> wrong, check the cited source before blaming the model — it is usually reporting a real
|
||||
> problem in the repo.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS DOES NOT DO ━━━
|
||||
|
||||
- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
|
||||
an empty key whitelist. Nothing in this folder writes a setting.
|
||||
- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
|
||||
per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
|
||||
- **It does not index code.** Headers and docs only.
|
||||
- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
|
||||
- **It is not required.** Every script runs identically with `AI_ENABLED=false`.
|
||||
|
||||
See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
|
||||
Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
|
||||
built before any of that is enabled.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCHEDULING ━━━
|
||||
|
||||
Not scheduled by default. When you want the index to track the repo automatically, the natural
|
||||
home is after a successful pull — the only moment the corpus actually changes:
|
||||
|
||||
```bash
|
||||
# master.conf → DAILY_MAINTENANCE_SCRIPTS, after git_pull_execute.sh
|
||||
"AI/ai_index.sh"
|
||||
```
|
||||
|
||||
`AI_INDEX_ON_PULL` exists in `master.conf` as the intended gate for wiring this into
|
||||
`git_pull_execute.sh` directly. It is not yet consumed by anything — the variable is reserved,
|
||||
not live.
|
||||
|
||||
An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
|
||||
and a pull that changed twelve files costs a few seconds.
|
||||
Executable
+244
@@ -0,0 +1,244 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ━━━ AI Retrieval Index Builder ━━━
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ==============================================================================================
|
||||
# Builds and refreshes the retrieval index over Varaverk's own documentation — every script
|
||||
# header, folder README and Manual, and the conf templates. Chunks each file on the section
|
||||
# boundaries the header convention already defines, embeds each chunk through Ollama, and
|
||||
# stores the vectors in SQLite for AI/ai_query.sh to search.
|
||||
#
|
||||
# The index is derived data. It is gitignored, host-local, and rebuildable from the repo in
|
||||
# minutes — nothing depends on it surviving.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
# Chunking mirrors the header convention rather than using a fixed token window:
|
||||
#
|
||||
# bash headers split on the six section names, then sub-split named-paragraph
|
||||
# safeguards and principles so one question finds one answer
|
||||
# PHP headers same six names plus the per-layer tails (EXPORTS, RENDERS, ...)
|
||||
# markdown split on headings
|
||||
# conf templates split on the ━━━ section rules
|
||||
#
|
||||
# Every chunk keeps its section name as a field, which is what lets a query about a safeguard
|
||||
# be steered toward OPERATIONAL SAFEGUARDS chunks before similarity is considered.
|
||||
#
|
||||
# Incremental by file mtime. A file whose mtime has not moved is skipped without being read,
|
||||
# so a routine refresh costs well under a second and a full rebuild costs a few minutes.
|
||||
#
|
||||
# Heavy lifting runs in Node — float vector maths and SQLite BLOBs are genuinely awkward in
|
||||
# bash. This follows the api_cache_writer.sh precedent: a bash shim owning config, gating,
|
||||
# locking and logging, in front of the language that fits the work.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tracked Files Only
|
||||
# Indexes exactly what `git ls-files` reports. Configurations/, State_Files/ and data/ are
|
||||
# gitignored, so it is structurally impossible for a credential to reach the index — the
|
||||
# files holding them were never in the repo. This must never become a filesystem walk: a
|
||||
# secret written into a vector cannot be rotated back out of it.
|
||||
#
|
||||
# The Index Is Disposable
|
||||
# Stored under DATA_DIR, gitignored, and never synced to a partner. Losing it costs one
|
||||
# rebuild. Nothing reads it as a source of truth — it points at files, and the files are
|
||||
# the truth.
|
||||
#
|
||||
# Documentation Is The Corpus, Not The Code
|
||||
# Script bodies are not indexed. The headers state intent and the code states mechanism;
|
||||
# for the questions this answers, intent retrieves far better and is far cheaper.
|
||||
#
|
||||
# Off By Default
|
||||
# Does nothing unless AI_ENABLED is true. A node with AI off never pays for this.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Fail-Closed Gate
|
||||
# Exits cleanly unless AI_ENABLED is exactly "true". Any other value, including unset,
|
||||
# means off.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Writes into DATA_DIR alongside other Varaverk state.
|
||||
#
|
||||
# Concurrency Lock
|
||||
# acquire_lock() prevents two indexers racing on the same database.
|
||||
#
|
||||
# Dependency Verification
|
||||
# Verifies node and the Ollama endpoint before touching the database. A missing dependency
|
||||
# is reported and exits non-zero rather than leaving a half-built index.
|
||||
#
|
||||
# Reachability Pre-flight
|
||||
# Probes the resolved Ollama URL with AI_CONNECT_TIMEOUT before starting. An unreachable
|
||||
# endpoint aborts immediately instead of failing once per batch across the whole corpus.
|
||||
#
|
||||
# Partial Failure Is Not Recorded As Success
|
||||
# A file whose chunks all failed to embed keeps its previous rows and its old mtime, so the
|
||||
# next run retries it. A run with any failed batch exits 3.
|
||||
#
|
||||
# Atomic Per-Run Write
|
||||
# All database changes commit in one transaction. An interrupted run leaves the previous
|
||||
# index intact rather than a partially rewritten one.
|
||||
#
|
||||
# Deleted Files Are Removed From The Index
|
||||
# A file that has left the repo has its chunks deleted, so retrieval cannot cite something
|
||||
# that no longer exists.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# AI_ENABLED
|
||||
# Master switch. Fail-closed — must be exactly "true".
|
||||
#
|
||||
# AI_INDEX_DB
|
||||
# SQLite index path. (shipped default: $DATA_DIR/ai_index.db)
|
||||
#
|
||||
# AI_INDEX_BATCH
|
||||
# Chunks per embed request. (shipped default: 32)
|
||||
#
|
||||
# AI_CONNECT_TIMEOUT
|
||||
# Seconds for the reachability probe. (shipped default: 5)
|
||||
#
|
||||
# AI_REQUEST_TIMEOUT
|
||||
# Seconds for a single embed batch. (shipped default: 240)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_OLLAMA_URL
|
||||
# This host's Ollama endpoint. Empty means no local Ollama.
|
||||
#
|
||||
# HOST*_OLLAMA_EMBED_MODEL
|
||||
# Embedding model. The generation model cannot embed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ai_index.sh
|
||||
# Incremental refresh — only files whose mtime moved are re-embedded.
|
||||
#
|
||||
# ai_index.sh --force
|
||||
# Full rebuild. Discards the existing index and re-embeds every chunk.
|
||||
#
|
||||
# ai_index.sh --dry-run
|
||||
# Report what would be indexed. Contacts nothing and writes nothing.
|
||||
#
|
||||
# ai_index.sh --status
|
||||
# Show index location, size, chunk counts by kind and section, and last build time.
|
||||
#
|
||||
# ai_index.sh --log
|
||||
# Verbose — per-batch embedding progress.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "${SCRIPT_DIR}/../load_config.sh"
|
||||
|
||||
detect_hosts
|
||||
|
||||
DRY_RUN=false; FORCE=false; STATUS=false; LOG=false
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
--force) FORCE=true ;;
|
||||
--status) STATUS=true ;;
|
||||
--log) LOG=true ;;
|
||||
*) echo "Unknown option: $arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
CLI="${SCRIPT_DIR}/lib/cli.js"
|
||||
DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
|
||||
|
||||
_url_var="${MY_ID}_OLLAMA_URL"
|
||||
_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
|
||||
OLLAMA_URL="${!_url_var:-}"
|
||||
EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
|
||||
|
||||
# ── Status ────────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$STATUS" == true ]]; then
|
||||
echo "$ICON_GEAR AI Index Status"
|
||||
echo " Enabled: ${AI_ENABLED:-false}"
|
||||
echo " Database: $DB"
|
||||
if [[ -f "$DB" ]]; then
|
||||
echo " Size: $(du -h "$DB" 2>/dev/null | cut -f1)"
|
||||
echo " Chunks: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null || echo '?')"
|
||||
echo " Files: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_files;' 2>/dev/null || echo '?')"
|
||||
_last=$(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='last_index';" 2>/dev/null)
|
||||
[[ -n "$_last" ]] && echo " Last built: $(date -d "@$_last" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)"
|
||||
echo " Model: $(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='embed_model';" 2>/dev/null || echo '?')"
|
||||
echo " By kind:"
|
||||
sqlite3 "$DB" "SELECT ' '||kind||': '||COUNT(*) FROM vv_chunks GROUP BY kind ORDER BY COUNT(*) DESC;" 2>/dev/null
|
||||
else
|
||||
echo " Database: not built yet"
|
||||
fi
|
||||
echo " Ollama: ${OLLAMA_URL:-<none local>}"
|
||||
echo " Embed: $EMBED_MODEL"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Gate ──────────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "${AI_ENABLED:-false}" != "true" ]]; then
|
||||
log "AI_ENABLED is not true — skipping index build"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == false && "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
command -v node >/dev/null 2>&1 || { error "node not found — required to build the index"; exit 1; }
|
||||
[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
|
||||
|
||||
if [[ -z "$OLLAMA_URL" ]]; then
|
||||
error "${MY_ID}_OLLAMA_URL is empty — no local Ollama to index against"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pre-flight: fail once, up front, rather than once per batch across the whole corpus.
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
|
||||
error "Ollama unreachable at $OLLAMA_URL"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Build ─────────────────────────────────────────────────────────────────────────────────────
|
||||
_args=(
|
||||
index
|
||||
"--root=${SCRIPTS_DIR}"
|
||||
"--db=${DB}"
|
||||
"--url=${OLLAMA_URL}"
|
||||
"--model=${EMBED_MODEL}"
|
||||
"--batch=${AI_INDEX_BATCH:-32}"
|
||||
"--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
|
||||
)
|
||||
[[ "$FORCE" == true ]] && _args+=(--force)
|
||||
[[ "$DRY_RUN" == true ]] && _args+=(--dry-run)
|
||||
[[ "$LOG" == false ]] && _args+=(--quiet)
|
||||
|
||||
log "$ICON_GEAR Building AI index → $DB"
|
||||
node --no-warnings "$CLI" "${_args[@]}"
|
||||
_rc=$?
|
||||
|
||||
case "$_rc" in
|
||||
0) log "$ICON_DONE AI index build complete" ;;
|
||||
3) warn "AI index built with some batches failed — those files will retry next run" ;;
|
||||
*) error "AI index build failed (exit $_rc)" ;;
|
||||
esac
|
||||
|
||||
exit "$_rc"
|
||||
Executable
+184
@@ -0,0 +1,184 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ━━━ AI Retrieval Query ━━━
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ==============================================================================================
|
||||
# Answers questions about Varaverk from Varaverk's own documentation. Embeds the question,
|
||||
# retrieves the closest chunks from the index AI/ai_index.sh built, and either prints them
|
||||
# directly or passes them to the generation model as grounding context.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
# Two modes over the same retrieval:
|
||||
#
|
||||
# --search print the matching chunks and their sources. No generation model involved,
|
||||
# so it is fast and its output is verbatim repo text.
|
||||
# (default) retrieve, then ask the generation model to answer strictly from what was
|
||||
# retrieved, citing each claim.
|
||||
#
|
||||
# Retrieval is steered by question shape. A question about what prevents something is pushed
|
||||
# toward OPERATIONAL SAFEGUARDS chunks, one about a variable toward CONFIGURATION, one asking
|
||||
# why toward DESIGN PRINCIPLES. This is a score boost, not a filter — a heuristic must not be
|
||||
# able to exclude the chunk that actually holds the answer.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Grounded Or Silent
|
||||
# The prompt instructs the model to answer only from retrieved context and to say what is
|
||||
# missing rather than fill the gap. This repo's conventions are frequently not the
|
||||
# conventional ones, and a confident generic answer about rsync or Docker is worse here
|
||||
# than no answer.
|
||||
#
|
||||
# Sources Are Always Shown
|
||||
# Every answer prints the chunks it drew on. An answer that cannot be traced back to a file
|
||||
# is not usable for changing anything.
|
||||
#
|
||||
# Search Is The Trustworthy Mode
|
||||
# --search returns repo text with nothing generated. When an answer matters, use it.
|
||||
#
|
||||
# Read-Only
|
||||
# Retrieves and answers. Nothing here writes conf, touches state, or runs another script.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Fail-Closed Gate
|
||||
# Exits cleanly unless AI_ENABLED is exactly "true".
|
||||
#
|
||||
# No Root Required
|
||||
# Reads the index and calls Ollama. Nothing it does needs privilege, so it does not ask for
|
||||
# any — this is the one AI script an ordinary user should be able to run.
|
||||
#
|
||||
# Missing Index Is Reported, Not Built
|
||||
# An absent index exits with guidance to run ai_index.sh. Building a corpus-wide index as a
|
||||
# side effect of a question would turn a two-second query into a several-minute one.
|
||||
#
|
||||
# Reachability Pre-flight
|
||||
# Probes Ollama with AI_CONNECT_TIMEOUT before embedding, so an unreachable endpoint fails
|
||||
# immediately with a clear message.
|
||||
#
|
||||
# Bounded Generation
|
||||
# The request is capped at AI_REQUEST_TIMEOUT. A wedged model cannot hang the caller.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# AI_ENABLED
|
||||
# Master switch. Fail-closed — must be exactly "true".
|
||||
#
|
||||
# AI_INDEX_DB
|
||||
# SQLite index to search. (shipped default: $DATA_DIR/ai_index.db)
|
||||
#
|
||||
# AI_SEARCH_K
|
||||
# Chunks retrieved per query. (shipped default: 8)
|
||||
#
|
||||
# AI_SEARCH_PER_FILE
|
||||
# Cap per file, so one document cannot fill the context. (shipped default: 3)
|
||||
#
|
||||
# AI_REQUEST_TIMEOUT
|
||||
# Seconds allowed for generation. (shipped default: 240)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_OLLAMA_URL / HOST*_OLLAMA_MODEL / HOST*_OLLAMA_EMBED_MODEL
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ai_query.sh "your question"
|
||||
# Retrieve and answer, with sources.
|
||||
#
|
||||
# ai_query.sh --search "your question"
|
||||
# Print matching chunks only. No generation.
|
||||
#
|
||||
# ai_query.sh --section=CONFIGURATION "your question"
|
||||
# Restrict retrieval to one header section.
|
||||
#
|
||||
# ai_query.sh --json "your question"
|
||||
# Machine-readable output for other scripts.
|
||||
#
|
||||
# ai_query.sh --status
|
||||
# Show index and endpoint state.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "${SCRIPT_DIR}/../load_config.sh"
|
||||
|
||||
detect_hosts
|
||||
|
||||
SEARCH_ONLY=false; JSON=false; STATUS=false; SECTION=""; QUERY=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--search) SEARCH_ONLY=true ;;
|
||||
--json) JSON=true ;;
|
||||
--status) STATUS=true ;;
|
||||
--section=*) SECTION="${arg#*=}" ;;
|
||||
--*) echo "Unknown option: $arg" >&2; exit 1 ;;
|
||||
*) QUERY="$arg" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
CLI="${SCRIPT_DIR}/lib/cli.js"
|
||||
DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
|
||||
|
||||
_url_var="${MY_ID}_OLLAMA_URL"
|
||||
_gen_var="${MY_ID}_OLLAMA_MODEL"
|
||||
_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
|
||||
OLLAMA_URL="${!_url_var:-}"
|
||||
GEN_MODEL="${!_gen_var:-}"
|
||||
EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
|
||||
|
||||
if [[ "$STATUS" == true ]]; then
|
||||
echo "$ICON_GEAR AI Query Status"
|
||||
echo " Enabled: ${AI_ENABLED:-false}"
|
||||
echo " Index: $DB $([[ -f "$DB" ]] && echo "($(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null) chunks)" || echo '(not built)')"
|
||||
echo " Ollama: ${OLLAMA_URL:-<none>}"
|
||||
echo " Generate: ${GEN_MODEL:-<unset>}"
|
||||
echo " Embed: $EMBED_MODEL"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "${AI_ENABLED:-false}" != "true" ]]; then
|
||||
echo "AI_ENABLED is not true — AI features are off" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ -z "$QUERY" ]] && { echo "usage: ai_query.sh [--search] [--section=NAME] \"your question\"" >&2; exit 1; }
|
||||
[[ -f "$DB" ]] || { error "No index at $DB — run AI/ai_index.sh first"; exit 1; }
|
||||
[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
|
||||
command -v node >/dev/null 2>&1 || { error "node not found"; exit 1; }
|
||||
[[ -z "$OLLAMA_URL" ]] && { error "${MY_ID}_OLLAMA_URL is empty"; exit 1; }
|
||||
|
||||
if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
|
||||
error "Ollama unreachable at $OLLAMA_URL"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
_args=("--db=${DB}" "--url=${OLLAMA_URL}" "--query=${QUERY}")
|
||||
[[ -n "$SECTION" ]] && _args+=("--section=${SECTION}")
|
||||
[[ "$JSON" == true ]] && _args+=(--json)
|
||||
|
||||
if [[ "$SEARCH_ONLY" == true ]]; then
|
||||
node --no-warnings "$CLI" search "${_args[@]}" \
|
||||
"--model=${EMBED_MODEL}" \
|
||||
"--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}"
|
||||
else
|
||||
[[ -z "$GEN_MODEL" ]] && { error "${MY_ID}_OLLAMA_MODEL is empty — needed for generation"; exit 1; }
|
||||
node --no-warnings "$CLI" ask "${_args[@]}" \
|
||||
"--model=${GEN_MODEL}" "--embed-model=${EMBED_MODEL}" \
|
||||
"--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}" \
|
||||
"--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
|
||||
fi
|
||||
+9
-7
@@ -122,13 +122,9 @@ async function buildIndex(opts) {
|
||||
|
||||
let chunks = [];
|
||||
try { chunks = chunkFile(abs, rel); } catch (e) { continue; }
|
||||
if (!chunks.length) {
|
||||
if (db) {
|
||||
db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(rel);
|
||||
db.prepare('INSERT OR REPLACE INTO vv_files VALUES (?,?,?,?)').run(rel, mtime, 0, now);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// A file that yields no chunks still gets a vv_files row so it is not re-chunked every
|
||||
// run. Queued rather than written here, so every database change lands in the single
|
||||
// commit below — a run interrupted mid-embed must leave the index exactly as it was.
|
||||
pending.push({ rel, mtime, chunks });
|
||||
}
|
||||
|
||||
@@ -166,6 +162,12 @@ async function buildIndex(opts) {
|
||||
db.exec('BEGIN');
|
||||
try {
|
||||
for (const f of pending) {
|
||||
// Nothing to index in this file at all — record it so it is not re-chunked next run.
|
||||
if (!f.chunks.length) {
|
||||
db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel);
|
||||
insF.run(f.rel, f.mtime, 0, now);
|
||||
continue;
|
||||
}
|
||||
const embedded = f.chunks.filter(c => c.__vec);
|
||||
// A file whose chunks all failed to embed keeps its previous rows and its old mtime,
|
||||
// so the next run retries it rather than recording a half-indexed file as current.
|
||||
|
||||
@@ -569,6 +569,20 @@
|
||||
HOSTN_LLDAP_USER="admin" # lldap admin username
|
||||
HOSTN_LLDAP_PASS="" # lldap admin password
|
||||
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Ollama / AI ───────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Per-host because only some nodes actually have a GPU. A node with an empty OLLAMA_URL is not
|
||||
# an error — it falls through to the resolver and uses another node's Ollama over Tailscale.
|
||||
|
||||
# ━━━ Ollama ━━━
|
||||
HOSTN_OLLAMA_URL="" # e.g. http://localhost:11434 — empty if no local Ollama
|
||||
HOSTN_OLLAMA_CONTAINER="Ollama" # for watchdog / restart lists
|
||||
HOSTN_OLLAMA_GPU_UUID="" # pins Ollama to one card on multi-GPU hosts
|
||||
HOSTN_OLLAMA_MODEL="qwen2.5-coder:14b" # generation
|
||||
HOSTN_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — the generation model cannot embed
|
||||
|
||||
# ━━━ Authelia ━━━
|
||||
HOSTN_AUTHELIA_CONFIG="/mnt/user/appdata-Fallback/Critical-Data/Authelia/configuration.yml"
|
||||
HOSTN_AUTHELIA_CONTAINER="Authelia"
|
||||
|
||||
@@ -1593,6 +1593,54 @@
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity better than crashing mid-check
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting move better than crashing mid-move
|
||||
|
||||
# ==============================================================================================
|
||||
# ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
|
||||
# script to function — every feature that can lean on AI has a complete non-AI path, and the
|
||||
# AI path is an enhancement layered on top. If Ollama is unreachable, callers proceed without it.
|
||||
#
|
||||
# AI_ENABLED is necessary but not sufficient. Every feature stays individually off until it has
|
||||
# earned it — narration for months before anything is allowed near a decision.
|
||||
|
||||
# ━━━ AI Master Switch ━━━
|
||||
# Fail-closed: anything other than the literal "true" means off.
|
||||
AI_ENABLED=false
|
||||
AI_CONNECT_TIMEOUT=5 # seconds — probe when resolving which node has Ollama
|
||||
AI_REQUEST_TIMEOUT=240 # seconds — must clear a cold model load
|
||||
AI_RESOLVE_CACHE_TTL=300 # seconds — don't re-probe the mesh every invocation
|
||||
AI_MAX_RETRIES=1 # AI is enhancement; do not retry hard
|
||||
|
||||
# ━━━ AI Retrieval Index ━━━
|
||||
# The RAG index over this repo's own headers and documentation. Regenerable in minutes and
|
||||
# gitignored — it is derived data, never a source of truth.
|
||||
#
|
||||
# Only git-tracked files are ever indexed. Configurations/, State_Files/ and data/ are
|
||||
# gitignored, which is what makes it structurally impossible for a credential to reach the
|
||||
# index: the files holding them were never in the repo. Do not "improve" this to a filesystem
|
||||
# walk — an embedded secret cannot be rotated out of a vector.
|
||||
AI_INDEX_DB="$DATA_DIR/ai_index.db"
|
||||
AI_INDEX_BATCH=32 # chunks per embed request
|
||||
AI_INDEX_ON_PULL=false # re-index after a successful git pull once AI is in use
|
||||
AI_SEARCH_K=8 # chunks retrieved per query
|
||||
AI_SEARCH_PER_FILE=3 # cap per file so one document cannot fill the context
|
||||
|
||||
# ━━━ AI Feature Toggles ━━━
|
||||
# Tier 1 is narration — it cannot change a decision. Tier 2 adds context to a decision a script
|
||||
# already made. Tier 3 assists a human. Enable in that order, and give each one weeks.
|
||||
AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration
|
||||
AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition
|
||||
AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgement calls
|
||||
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage
|
||||
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
|
||||
|
||||
# ━━━ AI Conf Write Access ━━━
|
||||
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
|
||||
# never credentials, never a container name. An empty whitelist means no writes regardless of
|
||||
# the toggle.
|
||||
AI_CONF_WRITE_ENABLED=false
|
||||
AI_CONF_WRITE_KEYS=()
|
||||
|
||||
# ==============================================================================================
|
||||
# ──────────────────────── End Of User Variables ───────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
+69
-2
@@ -1,7 +1,13 @@
|
||||
# Varaverk AI Integration — Design Notes
|
||||
|
||||
**Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no
|
||||
`AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives.
|
||||
**Status: retrieval is built; integration is not.** As of 2026-08-02 the `AI_*` and
|
||||
`HOST*_OLLAMA_*` variables exist in both confs and both templates, and `AI/` holds a working
|
||||
index and query path — see the RAG section at the end of this document and `AI/README-AI.md`.
|
||||
|
||||
Everything else below remains design only. **No Varaverk script consults AI.** Every
|
||||
`AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false with an empty whitelist, and
|
||||
host resolution across the mesh is specified but not implemented. Originally captured
|
||||
2026-08-01 so the reasoning survives.
|
||||
|
||||
Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for
|
||||
generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See
|
||||
@@ -490,3 +496,64 @@ new — the machinery already exists and was audited this session.
|
||||
possibly credentials pasted by a user.
|
||||
- Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the
|
||||
serialisation? Defer until an actual problem is felt.
|
||||
|
||||
---
|
||||
|
||||
## RAG — built 2026-08-02
|
||||
|
||||
Retrieval is live. `AI/` holds the implementation; `AI/README-AI.md` documents it in full. What
|
||||
follows is only what changed relative to the plan recorded above.
|
||||
|
||||
**Corpus is larger than estimated.** ~2,950 chunks across ~180 files, not the 690 header chunks
|
||||
projected. Sub-chunking is why — see below.
|
||||
|
||||
**Named-paragraph sub-chunking was necessary, and was not in the plan.** Section-level chunks
|
||||
alone were too coarse. `rsync.sh` documents fourteen safeguards in one 2.8k-char
|
||||
`OPERATIONAL SAFEGUARDS` block; a query about one of them scored 0.558, below unrelated chunks,
|
||||
because the other thirteen dominated the vector. Splitting on the named-paragraph titles the
|
||||
header convention already uses took the same query to 0.718 and first place. The parent section
|
||||
name is carried onto each sub-chunk so routing still works.
|
||||
|
||||
**Two chunker bugs worth remembering.** The last section in a header (RUNTIME MODES in bash,
|
||||
DEPENDS ON in a page) ran to EOF and swept up every unrelated comment in the file —
|
||||
`scheduler.php` alone produced an 11k-char chunk of unrelated inline comments. And title
|
||||
detection must require the *next* line to be indented; without that, any wrapped prose line
|
||||
became a spurious chunk boundary mid-sentence.
|
||||
|
||||
**Section routing is a boost, not a filter.** Intent detection is a heuristic and must not be
|
||||
able to exclude the chunk holding the answer. `--section=` forces a hard filter when wanted.
|
||||
|
||||
**Vectors arrive pre-normalised.** `nomic-embed-text` returns L2-normalised vectors (measured
|
||||
norm 1.0000001), so cosine is a plain dot product. No normalising step, no magnitude cache.
|
||||
|
||||
**`node:sqlite` over a native module.** Still flagged experimental, chosen because it needs no
|
||||
native compilation on Unraid. Acceptable because the index is disposable — if a Node upgrade
|
||||
breaks it, rebuild takes minutes. PHP reads the same float32 blobs with `unpack('f*', $blob)`
|
||||
when the UI needs them.
|
||||
|
||||
**Retrieval quality, measured.** 9/10 top-3 hit rate on known-answer questions; the tenth had
|
||||
the answer at ranks 2 and 3, so 10/10 for answer-present-in-context at k=6. Full retrieval plus
|
||||
generation runs about 43s warm.
|
||||
|
||||
### The finding that validated the whole thing
|
||||
|
||||
First real end-to-end question asked which variable controls the mover's grace window. The model
|
||||
answered `MOVER_STOP_TIMEOUT`, "defaults to 30 seconds", citing `mover_stop.sh › CONFIGURATION`.
|
||||
Variable correct; the 30 was wrong — the real value is 300. The model was quoting the header
|
||||
verbatim. **The header was stale.**
|
||||
|
||||
A sweep for the same pattern found six stale `(default: N)` claims across the repo — all
|
||||
corrected in the same pass. This is the operating principle for the folder:
|
||||
|
||||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||||
> wrong, check the cited source before blaming the model.
|
||||
|
||||
It also means the index is a documentation-drift detector, not only a question-answering tool.
|
||||
|
||||
### Still deliberately not built
|
||||
|
||||
Nothing consults this. Every `AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false
|
||||
with an empty key whitelist, and no watchdog, cleanup or fallback path calls it. Host resolution
|
||||
across the Tailscale mesh is specified above but not implemented — `ai_index.sh` and
|
||||
`ai_query.sh` currently require a local `HOST*_OLLAMA_URL` and fail with a clear message when it
|
||||
is empty, rather than silently probing the mesh.
|
||||
|
||||
Reference in New Issue
Block a user