Compare commits
2
Commits
ccc6c742ee
...
6a959fb5e4
| Author | SHA1 | Date | |
|---|---|---|---|
 Gmer4Lfe
|
6a959fb5e4 | ||
|
Gmer4Lfe
|
b00688bad1 |
@@ -0,0 +1,4 @@
|
||||
# The index is generated, host-specific, and regenerable in minutes. Never commit it.
|
||||
*.db
|
||||
*.db-wal
|
||||
*.db-shm
|
||||
+217
@@ -0,0 +1,217 @@
|
||||
# ━━━━━ AI ━━━━━
|
||||
|
||||
Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
|
||||
answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
|
||||
|
||||
Everything here is **off by default and optional**. Varaverk works exactly as well with
|
||||
`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
|
||||
|
||||
| File | Role |
|
||||
|------|------|
|
||||
| `ai_index.sh` | Build / refresh the retrieval index |
|
||||
| `ai_query.sh` | Ask a question, or search the index directly |
|
||||
| `lib/chunk.js` | Split repo files into retrieval units |
|
||||
| `lib/index.js` | Embed chunks, store vectors in SQLite |
|
||||
| `lib/search.js` | Embed a query, score it, rank results |
|
||||
| `lib/cli.js` | Argument bridge the bash entry points call |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHY THIS WORKS AT ALL ━━━
|
||||
|
||||
**Because the header audit came first.**
|
||||
|
||||
The single worst failure in naive retrieval is a chunk that contains half of one idea and half
|
||||
of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
|
||||
vector. That problem does not exist here, because every script in the repo carries the same six
|
||||
sections at exact, greppable boundaries:
|
||||
|
||||
```
|
||||
PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
|
||||
→ CONFIGURATION → RUNTIME MODES
|
||||
```
|
||||
|
||||
Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
|
||||
heuristics, no tuning.
|
||||
|
||||
**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
|
||||
does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
|
||||
`removeCompletedDownloads` being true on both arrs is intentional. Those live in
|
||||
`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
|
||||
more than an equivalent pile of source.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SECTION ROUTING ━━━
|
||||
|
||||
Every chunk stores its section name as its own column, so a question's *shape* can steer
|
||||
retrieval before similarity is even considered:
|
||||
|
||||
| Question shape | Steered toward |
|
||||
|----------------|----------------|
|
||||
| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
|
||||
| "which variable controls X" | `CONFIGURATION` |
|
||||
| "does this take --dry-run" | `RUNTIME MODES` |
|
||||
| "why is it built this way" | `DESIGN PRINCIPLES` |
|
||||
| "how does X work" | `OPERATIONAL MODEL` |
|
||||
|
||||
Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
|
||||
must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
|
||||
hard filter when you actually want one.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
|
||||
|
||||
Sections alone were not granular enough, and the failure was instructive.
|
||||
|
||||
`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
|
||||
`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
|
||||
correct chunk scored **0.558** — below unrelated chunks from other files — because the other
|
||||
thirteen safeguards dominated the vector.
|
||||
|
||||
The header convention writes each safeguard as a named paragraph: an unindented title, an
|
||||
indented body. Splitting on those titles took the same query to **0.718** and first place.
|
||||
|
||||
```
|
||||
Merge-Run Delete Interlock ← its own chunk
|
||||
--delete is applied only when pass 1 completed...
|
||||
```
|
||||
|
||||
The parent section name is carried onto every sub-chunk, so routing still works. The title
|
||||
detection requires the *next* line to be indented — without that check, any wrapped prose line
|
||||
became a spurious boundary mid-sentence.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT IS INDEXED ━━━
|
||||
|
||||
Roughly 2,900 chunks across ~180 files:
|
||||
|
||||
| Kind | Source |
|
||||
|------|--------|
|
||||
| `header` | Script headers — bash and PHP, sections and named paragraphs |
|
||||
| `readme` | Every `README-*.md` |
|
||||
| `manual` | Every `Manual-*.md` |
|
||||
| `template` | `Deployment/*.template` — the versioned conf schema |
|
||||
| `doc` | Top-level `README.md`, `Manual.md`, design notes |
|
||||
|
||||
**Script bodies are not indexed.** Headers state intent, code states mechanism; for the
|
||||
questions this answers, intent retrieves better and costs far less.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE SAFETY BOUNDARY ━━━
|
||||
|
||||
**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
|
||||
|
||||
`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
|
||||
was never in the repo to begin with. The index therefore describes the full conf schema (via the
|
||||
tracked templates, which carry all the explanatory comments) while structurally **never
|
||||
containing a secret**.
|
||||
|
||||
> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
|
||||
> averaged into a 768-dimension float cannot be found, let alone removed.
|
||||
|
||||
A live conf value the model genuinely needs should arrive through a tool call at query time,
|
||||
subject to redaction — never baked into a vector at index time.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW IT IS BUILT ━━━
|
||||
|
||||
**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
|
||||
per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
|
||||
back up, in exchange for nothing at this scale.
|
||||
|
||||
**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
|
||||
vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
|
||||
reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
|
||||
|
||||
**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
|
||||
no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
|
||||
|
||||
**Node for the maths, bash for everything else.** The bash entry points own configuration,
|
||||
gating, locking and logging exactly as every other Varaverk job does. Node owns only float
|
||||
vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
|
||||
|
||||
> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
|
||||
> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
|
||||
> regenerable in minutes — this is a disposable artefact, not a datastore.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ USAGE ━━━
|
||||
|
||||
```bash
|
||||
# Build (needs AI_ENABLED=true)
|
||||
bash AI/ai_index.sh # incremental
|
||||
bash AI/ai_index.sh --force # full rebuild
|
||||
bash AI/ai_index.sh --dry-run # what would be indexed; contacts nothing
|
||||
bash AI/ai_index.sh --status # size, counts, last build
|
||||
|
||||
# Ask
|
||||
bash AI/ai_query.sh "what stops rsync and the mover running at once"
|
||||
bash AI/ai_query.sh --search "why are the cache writers lockless"
|
||||
bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
|
||||
bash AI/ai_query.sh --json "..." # for other scripts
|
||||
```
|
||||
|
||||
**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
|
||||
When the answer matters, use it — or read the sources the generated answer cites.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
|
||||
|
||||
The generation prompt instructs the model to answer only from retrieved context and to say what
|
||||
is missing rather than fill the gap from general knowledge. That instruction matters here more
|
||||
than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
|
||||
generic answer about rsync, Docker or systemd is worse than no answer.
|
||||
|
||||
It works — and its faithfulness cut both ways on the first real test. Asked which variable
|
||||
controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
|
||||
seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
|
||||
real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
|
||||
|
||||
That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
|
||||
lesson is the operating principle for this whole folder:
|
||||
|
||||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||||
> wrong, check the cited source before blaming the model — it is usually reporting a real
|
||||
> problem in the repo.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS DOES NOT DO ━━━
|
||||
|
||||
- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
|
||||
an empty key whitelist. Nothing in this folder writes a setting.
|
||||
- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
|
||||
per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
|
||||
- **It does not index code.** Headers and docs only.
|
||||
- **It does not sync.** The index is host-local and gitignored. Each node builds its own.
|
||||
- **It is not required.** Every script runs identically with `AI_ENABLED=false`.
|
||||
|
||||
See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
|
||||
Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
|
||||
built before any of that is enabled.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCHEDULING ━━━
|
||||
|
||||
Not scheduled by default. When you want the index to track the repo automatically, the natural
|
||||
home is after a successful pull — the only moment the corpus actually changes:
|
||||
|
||||
```bash
|
||||
# master.conf → DAILY_MAINTENANCE_SCRIPTS, after git_pull_execute.sh
|
||||
"AI/ai_index.sh"
|
||||
```
|
||||
|
||||
`AI_INDEX_ON_PULL` exists in `master.conf` as the intended gate for wiring this into
|
||||
`git_pull_execute.sh` directly. It is not yet consumed by anything — the variable is reserved,
|
||||
not live.
|
||||
|
||||
An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
|
||||
and a pull that changed twelve files costs a few seconds.
|
||||
Executable
+244
@@ -0,0 +1,244 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ━━━ AI Retrieval Index Builder ━━━
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ==============================================================================================
|
||||
# Builds and refreshes the retrieval index over Varaverk's own documentation — every script
|
||||
# header, folder README and Manual, and the conf templates. Chunks each file on the section
|
||||
# boundaries the header convention already defines, embeds each chunk through Ollama, and
|
||||
# stores the vectors in SQLite for AI/ai_query.sh to search.
|
||||
#
|
||||
# The index is derived data. It is gitignored, host-local, and rebuildable from the repo in
|
||||
# minutes — nothing depends on it surviving.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
# Chunking mirrors the header convention rather than using a fixed token window:
|
||||
#
|
||||
# bash headers split on the six section names, then sub-split named-paragraph
|
||||
# safeguards and principles so one question finds one answer
|
||||
# PHP headers same six names plus the per-layer tails (EXPORTS, RENDERS, ...)
|
||||
# markdown split on headings
|
||||
# conf templates split on the ━━━ section rules
|
||||
#
|
||||
# Every chunk keeps its section name as a field, which is what lets a query about a safeguard
|
||||
# be steered toward OPERATIONAL SAFEGUARDS chunks before similarity is considered.
|
||||
#
|
||||
# Incremental by file mtime. A file whose mtime has not moved is skipped without being read,
|
||||
# so a routine refresh costs well under a second and a full rebuild costs a few minutes.
|
||||
#
|
||||
# Heavy lifting runs in Node — float vector maths and SQLite BLOBs are genuinely awkward in
|
||||
# bash. This follows the api_cache_writer.sh precedent: a bash shim owning config, gating,
|
||||
# locking and logging, in front of the language that fits the work.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tracked Files Only
|
||||
# Indexes exactly what `git ls-files` reports. Configurations/, State_Files/ and data/ are
|
||||
# gitignored, so it is structurally impossible for a credential to reach the index — the
|
||||
# files holding them were never in the repo. This must never become a filesystem walk: a
|
||||
# secret written into a vector cannot be rotated back out of it.
|
||||
#
|
||||
# The Index Is Disposable
|
||||
# Stored under DATA_DIR, gitignored, and never synced to a partner. Losing it costs one
|
||||
# rebuild. Nothing reads it as a source of truth — it points at files, and the files are
|
||||
# the truth.
|
||||
#
|
||||
# Documentation Is The Corpus, Not The Code
|
||||
# Script bodies are not indexed. The headers state intent and the code states mechanism;
|
||||
# for the questions this answers, intent retrieves far better and is far cheaper.
|
||||
#
|
||||
# Off By Default
|
||||
# Does nothing unless AI_ENABLED is true. A node with AI off never pays for this.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Fail-Closed Gate
|
||||
# Exits cleanly unless AI_ENABLED is exactly "true". Any other value, including unset,
|
||||
# means off.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Writes into DATA_DIR alongside other Varaverk state.
|
||||
#
|
||||
# Concurrency Lock
|
||||
# acquire_lock() prevents two indexers racing on the same database.
|
||||
#
|
||||
# Dependency Verification
|
||||
# Verifies node and the Ollama endpoint before touching the database. A missing dependency
|
||||
# is reported and exits non-zero rather than leaving a half-built index.
|
||||
#
|
||||
# Reachability Pre-flight
|
||||
# Probes the resolved Ollama URL with AI_CONNECT_TIMEOUT before starting. An unreachable
|
||||
# endpoint aborts immediately instead of failing once per batch across the whole corpus.
|
||||
#
|
||||
# Partial Failure Is Not Recorded As Success
|
||||
# A file whose chunks all failed to embed keeps its previous rows and its old mtime, so the
|
||||
# next run retries it. A run with any failed batch exits 3.
|
||||
#
|
||||
# Atomic Per-Run Write
|
||||
# All database changes commit in one transaction. An interrupted run leaves the previous
|
||||
# index intact rather than a partially rewritten one.
|
||||
#
|
||||
# Deleted Files Are Removed From The Index
|
||||
# A file that has left the repo has its chunks deleted, so retrieval cannot cite something
|
||||
# that no longer exists.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# AI_ENABLED
|
||||
# Master switch. Fail-closed — must be exactly "true".
|
||||
#
|
||||
# AI_INDEX_DB
|
||||
# SQLite index path. (shipped default: $DATA_DIR/ai_index.db)
|
||||
#
|
||||
# AI_INDEX_BATCH
|
||||
# Chunks per embed request. (shipped default: 32)
|
||||
#
|
||||
# AI_CONNECT_TIMEOUT
|
||||
# Seconds for the reachability probe. (shipped default: 5)
|
||||
#
|
||||
# AI_REQUEST_TIMEOUT
|
||||
# Seconds for a single embed batch. (shipped default: 240)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_OLLAMA_URL
|
||||
# This host's Ollama endpoint. Empty means no local Ollama.
|
||||
#
|
||||
# HOST*_OLLAMA_EMBED_MODEL
|
||||
# Embedding model. The generation model cannot embed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ai_index.sh
|
||||
# Incremental refresh — only files whose mtime moved are re-embedded.
|
||||
#
|
||||
# ai_index.sh --force
|
||||
# Full rebuild. Discards the existing index and re-embeds every chunk.
|
||||
#
|
||||
# ai_index.sh --dry-run
|
||||
# Report what would be indexed. Contacts nothing and writes nothing.
|
||||
#
|
||||
# ai_index.sh --status
|
||||
# Show index location, size, chunk counts by kind and section, and last build time.
|
||||
#
|
||||
# ai_index.sh --log
|
||||
# Verbose — per-batch embedding progress.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "${SCRIPT_DIR}/../load_config.sh"
|
||||
|
||||
detect_hosts
|
||||
|
||||
DRY_RUN=false; FORCE=false; STATUS=false; LOG=false
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
--force) FORCE=true ;;
|
||||
--status) STATUS=true ;;
|
||||
--log) LOG=true ;;
|
||||
*) echo "Unknown option: $arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
CLI="${SCRIPT_DIR}/lib/cli.js"
|
||||
DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
|
||||
|
||||
_url_var="${MY_ID}_OLLAMA_URL"
|
||||
_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
|
||||
OLLAMA_URL="${!_url_var:-}"
|
||||
EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
|
||||
|
||||
# ── Status ────────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$STATUS" == true ]]; then
|
||||
echo "$ICON_GEAR AI Index Status"
|
||||
echo " Enabled: ${AI_ENABLED:-false}"
|
||||
echo " Database: $DB"
|
||||
if [[ -f "$DB" ]]; then
|
||||
echo " Size: $(du -h "$DB" 2>/dev/null | cut -f1)"
|
||||
echo " Chunks: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null || echo '?')"
|
||||
echo " Files: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_files;' 2>/dev/null || echo '?')"
|
||||
_last=$(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='last_index';" 2>/dev/null)
|
||||
[[ -n "$_last" ]] && echo " Last built: $(date -d "@$_last" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)"
|
||||
echo " Model: $(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='embed_model';" 2>/dev/null || echo '?')"
|
||||
echo " By kind:"
|
||||
sqlite3 "$DB" "SELECT ' '||kind||': '||COUNT(*) FROM vv_chunks GROUP BY kind ORDER BY COUNT(*) DESC;" 2>/dev/null
|
||||
else
|
||||
echo " Database: not built yet"
|
||||
fi
|
||||
echo " Ollama: ${OLLAMA_URL:-<none local>}"
|
||||
echo " Embed: $EMBED_MODEL"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Gate ──────────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "${AI_ENABLED:-false}" != "true" ]]; then
|
||||
log "AI_ENABLED is not true — skipping index build"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == false && "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
command -v node >/dev/null 2>&1 || { error "node not found — required to build the index"; exit 1; }
|
||||
[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
|
||||
|
||||
if [[ -z "$OLLAMA_URL" ]]; then
|
||||
error "${MY_ID}_OLLAMA_URL is empty — no local Ollama to index against"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pre-flight: fail once, up front, rather than once per batch across the whole corpus.
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
|
||||
error "Ollama unreachable at $OLLAMA_URL"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Build ─────────────────────────────────────────────────────────────────────────────────────
|
||||
_args=(
|
||||
index
|
||||
"--root=${SCRIPTS_DIR}"
|
||||
"--db=${DB}"
|
||||
"--url=${OLLAMA_URL}"
|
||||
"--model=${EMBED_MODEL}"
|
||||
"--batch=${AI_INDEX_BATCH:-32}"
|
||||
"--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
|
||||
)
|
||||
[[ "$FORCE" == true ]] && _args+=(--force)
|
||||
[[ "$DRY_RUN" == true ]] && _args+=(--dry-run)
|
||||
[[ "$LOG" == false ]] && _args+=(--quiet)
|
||||
|
||||
log "$ICON_GEAR Building AI index → $DB"
|
||||
node --no-warnings "$CLI" "${_args[@]}"
|
||||
_rc=$?
|
||||
|
||||
case "$_rc" in
|
||||
0) log "$ICON_DONE AI index build complete" ;;
|
||||
3) warn "AI index built with some batches failed — those files will retry next run" ;;
|
||||
*) error "AI index build failed (exit $_rc)" ;;
|
||||
esac
|
||||
|
||||
exit "$_rc"
|
||||
Executable
+184
@@ -0,0 +1,184 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ━━━ AI Retrieval Query ━━━
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ==============================================================================================
|
||||
# Answers questions about Varaverk from Varaverk's own documentation. Embeds the question,
|
||||
# retrieves the closest chunks from the index AI/ai_index.sh built, and either prints them
|
||||
# directly or passes them to the generation model as grounding context.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
# Two modes over the same retrieval:
|
||||
#
|
||||
# --search print the matching chunks and their sources. No generation model involved,
|
||||
# so it is fast and its output is verbatim repo text.
|
||||
# (default) retrieve, then ask the generation model to answer strictly from what was
|
||||
# retrieved, citing each claim.
|
||||
#
|
||||
# Retrieval is steered by question shape. A question about what prevents something is pushed
|
||||
# toward OPERATIONAL SAFEGUARDS chunks, one about a variable toward CONFIGURATION, one asking
|
||||
# why toward DESIGN PRINCIPLES. This is a score boost, not a filter — a heuristic must not be
|
||||
# able to exclude the chunk that actually holds the answer.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Grounded Or Silent
|
||||
# The prompt instructs the model to answer only from retrieved context and to say what is
|
||||
# missing rather than fill the gap. This repo's conventions are frequently not the
|
||||
# conventional ones, and a confident generic answer about rsync or Docker is worse here
|
||||
# than no answer.
|
||||
#
|
||||
# Sources Are Always Shown
|
||||
# Every answer prints the chunks it drew on. An answer that cannot be traced back to a file
|
||||
# is not usable for changing anything.
|
||||
#
|
||||
# Search Is The Trustworthy Mode
|
||||
# --search returns repo text with nothing generated. When an answer matters, use it.
|
||||
#
|
||||
# Read-Only
|
||||
# Retrieves and answers. Nothing here writes conf, touches state, or runs another script.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Fail-Closed Gate
|
||||
# Exits cleanly unless AI_ENABLED is exactly "true".
|
||||
#
|
||||
# No Root Required
|
||||
# Reads the index and calls Ollama. Nothing it does needs privilege, so it does not ask for
|
||||
# any — this is the one AI script an ordinary user should be able to run.
|
||||
#
|
||||
# Missing Index Is Reported, Not Built
|
||||
# An absent index exits with guidance to run ai_index.sh. Building a corpus-wide index as a
|
||||
# side effect of a question would turn a two-second query into a several-minute one.
|
||||
#
|
||||
# Reachability Pre-flight
|
||||
# Probes Ollama with AI_CONNECT_TIMEOUT before embedding, so an unreachable endpoint fails
|
||||
# immediately with a clear message.
|
||||
#
|
||||
# Bounded Generation
|
||||
# The request is capped at AI_REQUEST_TIMEOUT. A wedged model cannot hang the caller.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# AI_ENABLED
|
||||
# Master switch. Fail-closed — must be exactly "true".
|
||||
#
|
||||
# AI_INDEX_DB
|
||||
# SQLite index to search. (shipped default: $DATA_DIR/ai_index.db)
|
||||
#
|
||||
# AI_SEARCH_K
|
||||
# Chunks retrieved per query. (shipped default: 8)
|
||||
#
|
||||
# AI_SEARCH_PER_FILE
|
||||
# Cap per file, so one document cannot fill the context. (shipped default: 3)
|
||||
#
|
||||
# AI_REQUEST_TIMEOUT
|
||||
# Seconds allowed for generation. (shipped default: 240)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_OLLAMA_URL / HOST*_OLLAMA_MODEL / HOST*_OLLAMA_EMBED_MODEL
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ai_query.sh "your question"
|
||||
# Retrieve and answer, with sources.
|
||||
#
|
||||
# ai_query.sh --search "your question"
|
||||
# Print matching chunks only. No generation.
|
||||
#
|
||||
# ai_query.sh --section=CONFIGURATION "your question"
|
||||
# Restrict retrieval to one header section.
|
||||
#
|
||||
# ai_query.sh --json "your question"
|
||||
# Machine-readable output for other scripts.
|
||||
#
|
||||
# ai_query.sh --status
|
||||
# Show index and endpoint state.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "${SCRIPT_DIR}/../load_config.sh"
|
||||
|
||||
detect_hosts
|
||||
|
||||
SEARCH_ONLY=false; JSON=false; STATUS=false; SECTION=""; QUERY=""
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--search) SEARCH_ONLY=true ;;
|
||||
--json) JSON=true ;;
|
||||
--status) STATUS=true ;;
|
||||
--section=*) SECTION="${arg#*=}" ;;
|
||||
--*) echo "Unknown option: $arg" >&2; exit 1 ;;
|
||||
*) QUERY="$arg" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
CLI="${SCRIPT_DIR}/lib/cli.js"
|
||||
DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
|
||||
|
||||
_url_var="${MY_ID}_OLLAMA_URL"
|
||||
_gen_var="${MY_ID}_OLLAMA_MODEL"
|
||||
_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
|
||||
OLLAMA_URL="${!_url_var:-}"
|
||||
GEN_MODEL="${!_gen_var:-}"
|
||||
EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
|
||||
|
||||
if [[ "$STATUS" == true ]]; then
|
||||
echo "$ICON_GEAR AI Query Status"
|
||||
echo " Enabled: ${AI_ENABLED:-false}"
|
||||
echo " Index: $DB $([[ -f "$DB" ]] && echo "($(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null) chunks)" || echo '(not built)')"
|
||||
echo " Ollama: ${OLLAMA_URL:-<none>}"
|
||||
echo " Generate: ${GEN_MODEL:-<unset>}"
|
||||
echo " Embed: $EMBED_MODEL"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "${AI_ENABLED:-false}" != "true" ]]; then
|
||||
echo "AI_ENABLED is not true — AI features are off" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ -z "$QUERY" ]] && { echo "usage: ai_query.sh [--search] [--section=NAME] \"your question\"" >&2; exit 1; }
|
||||
[[ -f "$DB" ]] || { error "No index at $DB — run AI/ai_index.sh first"; exit 1; }
|
||||
[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
|
||||
command -v node >/dev/null 2>&1 || { error "node not found"; exit 1; }
|
||||
[[ -z "$OLLAMA_URL" ]] && { error "${MY_ID}_OLLAMA_URL is empty"; exit 1; }
|
||||
|
||||
if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
|
||||
error "Ollama unreachable at $OLLAMA_URL"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
_args=("--db=${DB}" "--url=${OLLAMA_URL}" "--query=${QUERY}")
|
||||
[[ -n "$SECTION" ]] && _args+=("--section=${SECTION}")
|
||||
[[ "$JSON" == true ]] && _args+=(--json)
|
||||
|
||||
if [[ "$SEARCH_ONLY" == true ]]; then
|
||||
node --no-warnings "$CLI" search "${_args[@]}" \
|
||||
"--model=${EMBED_MODEL}" \
|
||||
"--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}"
|
||||
else
|
||||
[[ -z "$GEN_MODEL" ]] && { error "${MY_ID}_OLLAMA_MODEL is empty — needed for generation"; exit 1; }
|
||||
node --no-warnings "$CLI" ask "${_args[@]}" \
|
||||
"--model=${GEN_MODEL}" "--embed-model=${EMBED_MODEL}" \
|
||||
"--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}" \
|
||||
"--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
|
||||
fi
|
||||
+255
@@ -0,0 +1,255 @@
|
||||
'use strict';
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
// Chunker — turns repo files into retrieval units.
|
||||
//
|
||||
// The whole point of the header audit is that chunk boundaries are deterministic here. Bash
|
||||
// scripts split on their six section names, markdown on its headings, conf templates on their
|
||||
// ━━━ section rules. Nothing is split on a fixed token window, so no chunk ever contains half
|
||||
// of one idea and half of another.
|
||||
//
|
||||
// Every chunk carries its section name as its own field, because that is the metadata that
|
||||
// lets retrieval filter by question shape before it ever computes similarity.
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
const BASH_SECTIONS = [
|
||||
'PURPOSE', 'OPERATIONAL MODEL', 'DESIGN PRINCIPLES',
|
||||
'OPERATIONAL SAFEGUARDS', 'CONFIGURATION', 'RUNTIME MODES',
|
||||
];
|
||||
|
||||
// PHP headers reuse the first three names deliberately, then diverge per layer.
|
||||
const PHP_SECTIONS = [
|
||||
'PURPOSE', 'OPERATIONAL MODEL', 'DESIGN PRINCIPLES', 'OPERATIONAL SAFEGUARDS',
|
||||
'STATUS', 'EXPORTS', 'REQUEST CONTRACT', 'SIDE EFFECTS', 'RENDERS', 'DEPENDS ON',
|
||||
'CONFIGURATION',
|
||||
];
|
||||
|
||||
const MIN_CHARS = 40; // below this a chunk carries no retrievable meaning
|
||||
const MAX_CHARS = 6000; // above this, split on blank lines — protects the embed window
|
||||
|
||||
// Banner rules and box-drawing art are everywhere in this repo's headers. They carry no
|
||||
// meaning to embed, and a chunk that is mostly rule characters is pure noise in the index.
|
||||
// Measure a chunk by what is left after the decoration is removed, not by raw length.
|
||||
function meaningful(s) {
|
||||
return s.replace(/[═─━=_#\/*\s|+.-]/g, '').length;
|
||||
}
|
||||
const MIN_MEANINGFUL = 30;
|
||||
|
||||
function isRealHeading(h) {
|
||||
return !!h && /[A-Za-z0-9]/.test(h.replace(/[═─━=_]/g, ''));
|
||||
}
|
||||
|
||||
function stripPrefix(line, prefix) {
|
||||
// '# text' -> 'text' '// text' -> 'text'
|
||||
const re = new RegExp('^\\s*' + prefix + '\\s?');
|
||||
return line.replace(re, '');
|
||||
}
|
||||
|
||||
// ── Comment-header sectioning, shared by bash (#) and PHP (//) ────────────────────────────────
|
||||
function sectionsFromCommentHeader(text, prefix, names) {
|
||||
const lines = text.split('\n');
|
||||
const nameSet = new Set(names);
|
||||
const found = [];
|
||||
|
||||
// headerEnd matters as much as the section starts. The last section (RUNTIME MODES in bash,
|
||||
// DEPENDS ON in a page) would otherwise run to EOF and sweep up every unrelated comment in
|
||||
// the file — scheduler.php alone contributed an 11k-char chunk of unrelated inline comments.
|
||||
let headerEnd = lines.length;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const raw = lines[i];
|
||||
if (!new RegExp('^\\s*' + prefix).test(raw)) {
|
||||
// Header block ends at the first non-comment, non-blank line past the shebang.
|
||||
// '<?php' and '?>' bracket a PHP header block and are not the end of it.
|
||||
const t = raw.trim();
|
||||
if (t !== '' && !/^#!/.test(t) && t !== '<?php' && t !== '?>' && found.length) {
|
||||
headerEnd = i;
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const inner = stripPrefix(raw, prefix).trim();
|
||||
if (nameSet.has(inner)) found.push({ name: inner, start: i });
|
||||
}
|
||||
|
||||
const out = [];
|
||||
for (let k = 0; k < found.length; k++) {
|
||||
const start = found[k].start + 1;
|
||||
const end = k + 1 < found.length ? found[k + 1].start : headerEnd;
|
||||
const body = lines.slice(start, end)
|
||||
.filter(l => new RegExp('^\\s*' + prefix).test(l))
|
||||
.map(l => stripPrefix(l, prefix))
|
||||
// drop pure separator rules (════, ────, ━━━) — they carry no meaning
|
||||
.filter(l => !/^[\s═─━=_-]*$/.test(l) || l.trim() === '')
|
||||
.join('\n')
|
||||
.replace(/\n{3,}/g, '\n\n')
|
||||
.trim();
|
||||
if (meaningful(body) >= MIN_MEANINGFUL) {
|
||||
for (const p of splitNamedParagraphs(body))
|
||||
out.push({ section: found[k].name, title: p.title, content: p.content });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── Named-paragraph sub-chunking ──────────────────────────────────────────────────────────────
|
||||
// The header convention writes safeguards and principles as named paragraphs: an unindented
|
||||
// title line followed by an indented body. Embedding a whole section as one unit dilutes them —
|
||||
// rsync.sh's OPERATIONAL SAFEGUARDS holds eight distinct guarantees in 2.8k chars, and a query
|
||||
// about one of them scored below unrelated chunks because the other seven dominated the vector.
|
||||
// Splitting on the title lines is what makes a specific question find a specific answer.
|
||||
//
|
||||
// The section name is carried onto every sub-chunk, so section routing still works; the
|
||||
// paragraph title becomes the chunk's heading.
|
||||
function splitNamedParagraphs(body) {
|
||||
const lines = body.split('\n');
|
||||
const marks = [];
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const l = lines[i];
|
||||
if (!l.trim()) continue;
|
||||
if (/^\s/.test(l)) continue; // indented => body, not a title
|
||||
if (/^[-*•]/.test(l.trim())) continue; // list item, not a title
|
||||
if (l.trim().length > 80) continue; // a long line is prose, not a heading
|
||||
if (/[.:;,]$/.test(l.trim())) continue; // ends like a sentence
|
||||
|
||||
// The decisive signal: a real title is followed by an indented body. Wrapped prose is
|
||||
// followed by more unindented prose. Without this check, any short line in a paragraph
|
||||
// that happened to wrap became a spurious chunk boundary mid-sentence.
|
||||
let j = i + 1;
|
||||
while (j < lines.length && !lines[j].trim()) j++;
|
||||
if (j >= lines.length || !/^\s+\S/.test(lines[j])) continue;
|
||||
|
||||
marks.push(i);
|
||||
}
|
||||
// Fewer than two titles means this section is not written as named paragraphs — keep it whole.
|
||||
if (marks.length < 2) return [{ title: null, content: body }];
|
||||
|
||||
const out = [];
|
||||
if (marks[0] > 0) {
|
||||
const pre = lines.slice(0, marks[0]).join('\n').trim();
|
||||
if (meaningful(pre) >= MIN_MEANINGFUL) out.push({ title: null, content: pre });
|
||||
}
|
||||
for (let k = 0; k < marks.length; k++) {
|
||||
const start = marks[k];
|
||||
const end = k + 1 < marks.length ? marks[k + 1] : lines.length;
|
||||
const title = lines[start].trim();
|
||||
const content = lines.slice(start, end).join('\n').trim();
|
||||
if (meaningful(content) >= MIN_MEANINGFUL) out.push({ title, content });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── Markdown: split on ## headings, keep the heading with its body ─────────────────────────────
|
||||
function sectionsFromMarkdown(text) {
|
||||
const lines = text.split('\n');
|
||||
const marks = [];
|
||||
let fence = false;
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (/^\s*```/.test(lines[i])) { fence = !fence; continue; }
|
||||
if (fence) continue;
|
||||
if (/^#{1,3}\s+\S/.test(lines[i])) marks.push(i);
|
||||
}
|
||||
if (!marks.length) return [{ heading: null, content: text.trim() }];
|
||||
|
||||
const out = [];
|
||||
// preamble before the first heading
|
||||
if (marks[0] > 0) {
|
||||
const pre = lines.slice(0, marks[0]).join('\n').trim();
|
||||
if (meaningful(pre) >= MIN_MEANINGFUL) out.push({ heading: null, content: pre });
|
||||
}
|
||||
for (let k = 0; k < marks.length; k++) {
|
||||
const start = marks[k];
|
||||
const end = k + 1 < marks.length ? marks[k + 1] : lines.length;
|
||||
const heading = lines[start].replace(/^#+\s*/, '').replace(/[━─═]+/g, '').trim();
|
||||
const content = lines.slice(start, end).join('\n').trim();
|
||||
if (meaningful(content) >= MIN_MEANINGFUL) out.push({ heading: isRealHeading(heading) ? heading : null, content });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ── Conf templates: split on the ━━━ / ── section rules ───────────────────────────────────────
|
||||
function sectionsFromConfTemplate(text) {
|
||||
const lines = text.split('\n');
|
||||
const marks = [];
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const m = lines[i].match(/^#\s*[━─]{2,}\s*(.+?)\s*[━─]{2,}\s*$/);
|
||||
if (m && isRealHeading(m[1])) marks.push({ i, name: m[1].trim() });
|
||||
}
|
||||
if (!marks.length) return [];
|
||||
|
||||
const out = [];
|
||||
for (let k = 0; k < marks.length; k++) {
|
||||
const start = marks[k].i;
|
||||
const end = k + 1 < marks.length ? marks[k + 1].i : lines.length;
|
||||
const content = lines.slice(start, end).join('\n').replace(/\n{3,}/g, '\n\n').trim();
|
||||
if (meaningful(content) >= MIN_MEANINGFUL) out.push({ heading: marks[k].name, content });
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Oversized chunks split on blank lines rather than mid-sentence.
|
||||
function capSize(chunks) {
|
||||
const out = [];
|
||||
for (const c of chunks) {
|
||||
if (c.content.length <= MAX_CHARS) { out.push(c); continue; }
|
||||
const paras = c.content.split(/\n\s*\n/);
|
||||
let buf = [], len = 0, part = 1;
|
||||
const flush = () => {
|
||||
if (!buf.length) return;
|
||||
out.push({ ...c, content: buf.join('\n\n'), part: part++ });
|
||||
buf = []; len = 0;
|
||||
};
|
||||
for (const p of paras) {
|
||||
if (len + p.length > MAX_CHARS && buf.length) flush();
|
||||
buf.push(p); len += p.length + 2;
|
||||
}
|
||||
flush();
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function classify(rel) {
|
||||
const base = path.basename(rel);
|
||||
if (rel.startsWith('Deployment/') && rel.endsWith('.template')) return 'template';
|
||||
if (base.endsWith('.md')) {
|
||||
if (base.startsWith('Manual')) return 'manual';
|
||||
if (base.startsWith('README') || base === 'README.md') return 'readme';
|
||||
return 'doc';
|
||||
}
|
||||
if (base.endsWith('.sh')) return 'header';
|
||||
if (base.endsWith('.php')) return 'header';
|
||||
return 'other';
|
||||
}
|
||||
|
||||
function chunkFile(absPath, rel) {
|
||||
const text = fs.readFileSync(absPath, 'utf8');
|
||||
const kind = classify(rel);
|
||||
let raw = [];
|
||||
|
||||
if (kind === 'header' && rel.endsWith('.sh')) {
|
||||
raw = sectionsFromCommentHeader(text, '#', BASH_SECTIONS)
|
||||
.map(s => ({ section: s.section, heading: s.title || null, content: s.content }));
|
||||
} else if (kind === 'header' && rel.endsWith('.php')) {
|
||||
raw = sectionsFromCommentHeader(text, '//', PHP_SECTIONS)
|
||||
.map(s => ({ section: s.section, heading: s.title || null, content: s.content }));
|
||||
} else if (kind === 'template') {
|
||||
raw = sectionsFromConfTemplate(text)
|
||||
.map(s => ({ section: null, heading: s.heading, content: s.content }));
|
||||
} else if (kind === 'readme' || kind === 'manual' || kind === 'doc') {
|
||||
raw = sectionsFromMarkdown(text)
|
||||
.map(s => ({ section: null, heading: s.heading, content: s.content }));
|
||||
}
|
||||
|
||||
return capSize(raw).map(c => ({
|
||||
path: rel,
|
||||
kind,
|
||||
section: c.section || null,
|
||||
heading: c.heading || null,
|
||||
part: c.part || null,
|
||||
content: c.content,
|
||||
}));
|
||||
}
|
||||
|
||||
module.exports = { chunkFile, classify, BASH_SECTIONS, PHP_SECTIONS };
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
'use strict';
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
// CLI bridge — the thin layer the bash entry points call.
|
||||
//
|
||||
// The bash scripts own configuration, gating, locking and logging, exactly as they do for every
|
||||
// other Varaverk job. This file owns only the work that is genuinely awkward in bash: float
|
||||
// vector math and SQLite BLOBs. That split follows the existing api_cache_writer.sh precedent —
|
||||
// a bash shim in front of the language that fits the task.
|
||||
//
|
||||
// Every value arrives as an argument or an environment variable read by the caller. This file
|
||||
// never reads a conf file itself, so there is exactly one place that decides what the settings
|
||||
// are.
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
const { buildIndex } = require('./index.js');
|
||||
const { search } = require('./search.js');
|
||||
|
||||
function arg(name, dflt) {
|
||||
const p = `--${name}=`;
|
||||
const hit = process.argv.find(a => a.startsWith(p));
|
||||
return hit ? hit.slice(p.length) : dflt;
|
||||
}
|
||||
function flag(name) {
|
||||
return process.argv.includes(`--${name}`);
|
||||
}
|
||||
|
||||
function fail(msg, code = 1) {
|
||||
console.error(msg);
|
||||
process.exit(code);
|
||||
}
|
||||
|
||||
async function cmdIndex() {
|
||||
const root = arg('root');
|
||||
const db = arg('db');
|
||||
const url = arg('url');
|
||||
const model = arg('model', 'nomic-embed-text');
|
||||
if (!root || !db || !url) fail('index: --root, --db and --url are required');
|
||||
|
||||
const quiet = flag('quiet');
|
||||
let stats;
|
||||
try {
|
||||
stats = await buildIndex({
|
||||
root, dbPath: db, url, model,
|
||||
batch: parseInt(arg('batch', '32'), 10),
|
||||
timeout: parseInt(arg('timeout', '120000'), 10),
|
||||
force: flag('force'),
|
||||
dryRun: flag('dry-run'),
|
||||
onProgress: p => {
|
||||
if (p.error) console.error(`embed batch failed: ${p.error}`);
|
||||
else if (!quiet && p.done % 320 === 0) console.log(` embedded ${p.done}/${p.total}`);
|
||||
},
|
||||
});
|
||||
} catch (e) {
|
||||
fail(`index failed: ${e.message}`, 2);
|
||||
}
|
||||
|
||||
if (flag('json')) { console.log(JSON.stringify(stats)); return; }
|
||||
if (stats.dryRun) {
|
||||
console.log(`DRY RUN — ${stats.files} file(s) would be indexed, ${stats.chunks} chunk(s) embedded`);
|
||||
console.log(` ${stats.skipped} unchanged, ${stats.removed} stale entr(ies) would be dropped`);
|
||||
return;
|
||||
}
|
||||
console.log(`indexed ${stats.files} file(s), ${stats.chunks} chunk(s) embedded`);
|
||||
console.log(` ${stats.skipped} unchanged, ${stats.removed} removed, ${stats.failed} failed`);
|
||||
console.log(` index now holds ${stats.total} chunk(s)`);
|
||||
// A partial index is usable but not complete — say so in the exit code so a caller can act.
|
||||
if (stats.failed) process.exit(3);
|
||||
}
|
||||
|
||||
async function cmdSearch() {
|
||||
const db = arg('db');
|
||||
const url = arg('url');
|
||||
const model = arg('model', 'nomic-embed-text');
|
||||
const q = arg('query');
|
||||
if (!db || !url || !q) fail('search: --db, --url and --query are required');
|
||||
|
||||
let r;
|
||||
try {
|
||||
r = await search({
|
||||
dbPath: db, url, embedModel: model, query: q,
|
||||
k: parseInt(arg('k', '8'), 10),
|
||||
perFile: parseInt(arg('per-file', '3'), 10),
|
||||
section: arg('section', null),
|
||||
kind: arg('kind', null),
|
||||
});
|
||||
} catch (e) {
|
||||
fail(`search failed: ${e.message}`, 2);
|
||||
}
|
||||
|
||||
if (flag('json')) { console.log(JSON.stringify(r)); return; }
|
||||
if (!r.results.length) { console.log('no matches'); return; }
|
||||
if (r.intents.length) console.log(`intent: ${r.intents.join(', ')}\n`);
|
||||
for (const x of r.results) {
|
||||
const label = x.heading || x.section || '-';
|
||||
console.log(`── ${x.score.toFixed(3)} ${x.path} [${x.section || x.kind}] ${label}`);
|
||||
console.log(x.content.split('\n').map(l => ' ' + l).join('\n'));
|
||||
console.log('');
|
||||
}
|
||||
}
|
||||
|
||||
// Retrieval + generation. The prompt is built here so the context block and the instructions
|
||||
// stay in one reviewable place.
|
||||
async function cmdAsk() {
|
||||
const db = arg('db');
|
||||
const url = arg('url');
|
||||
const embed = arg('embed-model', 'nomic-embed-text');
|
||||
const gen = arg('model');
|
||||
const q = arg('query');
|
||||
const timeout = parseInt(arg('timeout', '240000'), 10);
|
||||
if (!db || !url || !gen || !q) fail('ask: --db, --url, --model and --query are required');
|
||||
|
||||
let r;
|
||||
try {
|
||||
r = await search({
|
||||
dbPath: db, url, embedModel: embed, query: q,
|
||||
k: parseInt(arg('k', '6'), 10), perFile: parseInt(arg('per-file', '2'), 10),
|
||||
});
|
||||
} catch (e) {
|
||||
fail(`retrieval failed: ${e.message}`, 2);
|
||||
}
|
||||
if (!r.results.length) fail('no relevant context found in the index', 4);
|
||||
|
||||
const context = r.results.map((x, i) => {
|
||||
const label = [x.path, x.section, x.heading].filter(Boolean).join(' › ');
|
||||
return `[${i + 1}] ${label}\n${x.content}`;
|
||||
}).join('\n\n');
|
||||
|
||||
const prompt =
|
||||
`You are answering questions about Varaverk, a two-server self-healing home media ecosystem.
|
||||
|
||||
Answer ONLY from the context below. If the context does not contain the answer, say so plainly
|
||||
and name what is missing — do not fill the gap from general knowledge about Linux, Docker or
|
||||
rsync, because this system's conventions are frequently not the conventional ones.
|
||||
|
||||
Cite the source of each claim as [n]. Be concise and concrete.
|
||||
|
||||
CONTEXT
|
||||
${context}
|
||||
|
||||
QUESTION
|
||||
${q}
|
||||
|
||||
ANSWER`;
|
||||
|
||||
let res;
|
||||
try {
|
||||
res = await fetch(`${url.replace(/\/$/, '')}/api/generate`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: gen, prompt, stream: false,
|
||||
options: { temperature: 0.2, num_ctx: 8192 },
|
||||
}),
|
||||
signal: AbortSignal.timeout(timeout),
|
||||
});
|
||||
} catch (e) {
|
||||
fail(`generation failed: ${e.message}`, 2);
|
||||
}
|
||||
if (!res.ok) fail(`generation HTTP ${res.status}`, 2);
|
||||
const j = await res.json();
|
||||
|
||||
if (flag('json')) {
|
||||
console.log(JSON.stringify({ answer: j.response, sources: r.results.map(x => ({ path: x.path, section: x.section, heading: x.heading, score: x.score })) }));
|
||||
return;
|
||||
}
|
||||
console.log((j.response || '').trim());
|
||||
console.log('\nSources:');
|
||||
r.results.forEach((x, i) => {
|
||||
console.log(` [${i + 1}] ${[x.path, x.section, x.heading].filter(Boolean).join(' › ')}`);
|
||||
});
|
||||
}
|
||||
|
||||
const cmd = process.argv[2];
|
||||
const table = { index: cmdIndex, search: cmdSearch, ask: cmdAsk };
|
||||
if (!table[cmd]) fail(`usage: cli.js <index|search|ask> [--flags]`);
|
||||
table[cmd]().catch(e => fail(e.message, 2));
|
||||
+201
@@ -0,0 +1,201 @@
|
||||
'use strict';
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
// Indexer — chunk the repo, embed each chunk, store vectors in SQLite.
|
||||
//
|
||||
// Incremental by file mtime: a file whose mtime has not moved since its last index is skipped
|
||||
// entirely, so a routine re-index costs seconds rather than re-embedding the whole corpus.
|
||||
//
|
||||
// Vectors are stored as raw little-endian float32 BLOBs. nomic-embed-text returns L2-normalised
|
||||
// vectors, so cosine similarity is a plain dot product at query time — no normalising, no
|
||||
// magnitude cache. PHP can read the same blobs with unpack('f*', $blob) when the UI needs them.
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { execSync } = require('child_process');
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const { chunkFile, classify } = require('./chunk.js');
|
||||
|
||||
const SCHEMA = `
|
||||
CREATE TABLE IF NOT EXISTS vv_files (
|
||||
path TEXT PRIMARY KEY,
|
||||
mtime INTEGER NOT NULL,
|
||||
chunks INTEGER NOT NULL,
|
||||
indexed INTEGER NOT NULL
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS vv_chunks (
|
||||
id INTEGER PRIMARY KEY,
|
||||
path TEXT NOT NULL,
|
||||
kind TEXT NOT NULL,
|
||||
section TEXT,
|
||||
heading TEXT,
|
||||
part INTEGER,
|
||||
content TEXT NOT NULL,
|
||||
vector BLOB NOT NULL,
|
||||
indexed INTEGER NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_path ON vv_chunks(path);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_section ON vv_chunks(section);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_kind ON vv_chunks(kind);
|
||||
CREATE TABLE IF NOT EXISTS vv_meta (k TEXT PRIMARY KEY, v TEXT);
|
||||
`;
|
||||
|
||||
function openDb(dbPath) {
|
||||
fs.mkdirSync(path.dirname(dbPath), { recursive: true });
|
||||
const db = new DatabaseSync(dbPath);
|
||||
db.exec('PRAGMA journal_mode = WAL;');
|
||||
db.exec('PRAGMA synchronous = NORMAL;');
|
||||
db.exec(SCHEMA);
|
||||
return db;
|
||||
}
|
||||
|
||||
// Only ever index what git tracks. Configurations/, State_Files/ and data/ are gitignored, which
|
||||
// is what makes it structurally impossible for a credential to reach the index — the files that
|
||||
// hold them were never in the repo. Do not replace this with a filesystem walk.
|
||||
function trackedFiles(root) {
|
||||
return execSync('git ls-files', { cwd: root, maxBuffer: 1 << 26 })
|
||||
.toString().trim().split('\n')
|
||||
.filter(Boolean)
|
||||
.filter(f => classify(f) !== 'other');
|
||||
}
|
||||
|
||||
async function embedBatch(url, model, inputs, timeoutMs) {
|
||||
const ctl = AbortSignal.timeout(timeoutMs);
|
||||
const res = await fetch(`${url.replace(/\/$/, '')}/api/embed`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ model, input: inputs }),
|
||||
signal: ctl,
|
||||
});
|
||||
if (!res.ok) throw new Error(`embed HTTP ${res.status}: ${(await res.text()).slice(0, 200)}`);
|
||||
const j = await res.json();
|
||||
if (!j.embeddings || j.embeddings.length !== inputs.length)
|
||||
throw new Error(`embed returned ${j.embeddings ? j.embeddings.length : 0} of ${inputs.length}`);
|
||||
return j.embeddings;
|
||||
}
|
||||
|
||||
function toBlob(vec) {
|
||||
return Buffer.from(Float32Array.from(vec).buffer);
|
||||
}
|
||||
|
||||
async function buildIndex(opts) {
|
||||
const {
|
||||
root, dbPath, url, model,
|
||||
batch = 32, timeout = 120000, force = false, dryRun = false,
|
||||
onProgress = () => {},
|
||||
} = opts;
|
||||
|
||||
const db = dryRun ? null : openDb(dbPath);
|
||||
const now = Math.floor(Date.now() / 1000);
|
||||
|
||||
const known = new Map();
|
||||
if (db) for (const r of db.prepare('SELECT path, mtime FROM vv_files').all()) known.set(r.path, r.mtime);
|
||||
|
||||
const files = trackedFiles(root);
|
||||
const present = new Set(files);
|
||||
|
||||
// Files that left the repo must leave the index with them.
|
||||
let removed = 0;
|
||||
if (db && !force) {
|
||||
for (const p of known.keys()) {
|
||||
if (!present.has(p)) {
|
||||
db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(p);
|
||||
db.prepare('DELETE FROM vv_files WHERE path = ?').run(p);
|
||||
removed++;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (db && force) { db.exec('DELETE FROM vv_chunks; DELETE FROM vv_files;'); }
|
||||
|
||||
// ── Collect the chunks that actually need embedding ───────────────────────────────────────
|
||||
const pending = [];
|
||||
let skipped = 0, scanned = 0;
|
||||
|
||||
for (const rel of files) {
|
||||
const abs = path.join(root, rel);
|
||||
let st;
|
||||
try { st = fs.statSync(abs); } catch { continue; }
|
||||
const mtime = Math.floor(st.mtimeMs / 1000);
|
||||
scanned++;
|
||||
|
||||
if (!force && known.has(rel) && known.get(rel) === mtime) { skipped++; continue; }
|
||||
|
||||
let chunks = [];
|
||||
try { chunks = chunkFile(abs, rel); } catch (e) { continue; }
|
||||
// A file that yields no chunks still gets a vv_files row so it is not re-chunked every
|
||||
// run. Queued rather than written here, so every database change lands in the single
|
||||
// commit below — a run interrupted mid-embed must leave the index exactly as it was.
|
||||
pending.push({ rel, mtime, chunks });
|
||||
}
|
||||
|
||||
const totalChunks = pending.reduce((n, f) => n + f.chunks.length, 0);
|
||||
if (dryRun) {
|
||||
return { dryRun: true, scanned, skipped, removed, files: pending.length, chunks: totalChunks };
|
||||
}
|
||||
|
||||
// ── Embed in batches, write per file so an interrupted run leaves a consistent index ───────
|
||||
const flat = [];
|
||||
for (const f of pending) for (const c of f.chunks) flat.push({ f, c });
|
||||
|
||||
let done = 0, failed = 0;
|
||||
for (let i = 0; i < flat.length; i += batch) {
|
||||
const slice = flat.slice(i, i + batch);
|
||||
const inputs = slice.map(x => x.c.content);
|
||||
let vecs;
|
||||
try {
|
||||
vecs = await embedBatch(url, model, inputs, timeout);
|
||||
} catch (e) {
|
||||
failed += slice.length;
|
||||
onProgress({ done, total: flat.length, error: e.message });
|
||||
continue;
|
||||
}
|
||||
slice.forEach((x, k) => { x.c.__vec = vecs[k]; });
|
||||
done += slice.length;
|
||||
onProgress({ done, total: flat.length });
|
||||
}
|
||||
|
||||
const ins = db.prepare(
|
||||
'INSERT INTO vv_chunks (path,kind,section,heading,part,content,vector,indexed) VALUES (?,?,?,?,?,?,?,?)'
|
||||
);
|
||||
const insF = db.prepare('INSERT OR REPLACE INTO vv_files VALUES (?,?,?,?)');
|
||||
|
||||
db.exec('BEGIN');
|
||||
try {
|
||||
for (const f of pending) {
|
||||
// Nothing to index in this file at all — record it so it is not re-chunked next run.
|
||||
if (!f.chunks.length) {
|
||||
db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel);
|
||||
insF.run(f.rel, f.mtime, 0, now);
|
||||
continue;
|
||||
}
|
||||
const embedded = f.chunks.filter(c => c.__vec);
|
||||
// A file whose chunks all failed to embed keeps its previous rows and its old mtime,
|
||||
// so the next run retries it rather than recording a half-indexed file as current.
|
||||
if (!embedded.length) continue;
|
||||
db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel);
|
||||
for (const c of embedded) {
|
||||
ins.run(c.path, c.kind, c.section, c.heading, c.part, c.content, toBlob(c.__vec), now);
|
||||
}
|
||||
insF.run(f.rel, f.mtime, embedded.length, now);
|
||||
}
|
||||
db.prepare('INSERT OR REPLACE INTO vv_meta VALUES (?,?)').run('embed_model', model);
|
||||
db.prepare('INSERT OR REPLACE INTO vv_meta VALUES (?,?)').run('last_index', String(now));
|
||||
db.prepare('INSERT OR REPLACE INTO vv_meta VALUES (?,?)').run('dims', '768');
|
||||
db.exec('COMMIT');
|
||||
} catch (e) {
|
||||
db.exec('ROLLBACK');
|
||||
throw e;
|
||||
}
|
||||
|
||||
const stats = {
|
||||
scanned, skipped, removed,
|
||||
files: pending.length,
|
||||
chunks: done,
|
||||
failed,
|
||||
total: db.prepare('SELECT COUNT(*) n FROM vv_chunks').get().n,
|
||||
};
|
||||
db.close();
|
||||
return stats;
|
||||
}
|
||||
|
||||
module.exports = { buildIndex, openDb, toBlob, trackedFiles };
|
||||
@@ -0,0 +1,121 @@
|
||||
'use strict';
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
// Search — embed a question, score it against the index, return the best chunks.
|
||||
//
|
||||
// nomic-embed-text returns L2-normalised vectors, so cosine similarity is a plain dot product.
|
||||
// At this corpus size (~1.7k chunks x 768 dims) that is a couple of million multiply-adds —
|
||||
// under a millisecond, with no vector database and no index structure to maintain.
|
||||
//
|
||||
// Section routing is the payoff from the header audit. Every chunk knows whether it is a
|
||||
// PURPOSE, a DESIGN PRINCIPLES, an OPERATIONAL SAFEGUARDS and so on, so a question's shape can
|
||||
// steer retrieval before similarity is even considered. It is applied as a score boost rather
|
||||
// than a hard filter — intent detection is a heuristic, and a heuristic should not be able to
|
||||
// exclude the one chunk that actually holds the answer.
|
||||
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
||||
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
|
||||
// Question shape → the section most likely to answer it.
|
||||
const INTENTS = [
|
||||
{ section: 'OPERATIONAL SAFEGUARDS',
|
||||
re: /\b(safe|safety|guard|protect|prevent|fail|failure|abort|refuse|lock|root|timeout|dry.?run|what stops|what happens if|race|corrupt|data.?loss)\b/i },
|
||||
{ section: 'CONFIGURATION',
|
||||
re: /\b(variable|var|setting|conf|config|threshold|toggle|which key|what controls|where is .* set|default value|env)\b/i },
|
||||
{ section: 'RUNTIME MODES',
|
||||
re: /\b(flag|argument|option|--\w+|how do i run|invoke|cli|command line|status mode|usage)\b/i },
|
||||
{ section: 'DESIGN PRINCIPLES',
|
||||
re: /\b(why|rationale|reason|design|decision|deliberate|intentional|on purpose|trade.?off|chose|approach)\b/i },
|
||||
{ section: 'OPERATIONAL MODEL',
|
||||
re: /\b(how does .* work|flow|sequence|order|tier|lifecycle|state machine|when does)\b/i },
|
||||
{ section: 'EXPORTS',
|
||||
re: /\b(function|export|api surface|what does .* provide|helper|vv_\w+)\b/i },
|
||||
{ section: 'PURPOSE',
|
||||
re: /\b(what is|what does .* do|purpose|responsible for|job of)\b/i },
|
||||
];
|
||||
|
||||
const SECTION_BOOST = 0.06; // enough to reorder near-ties, not enough to beat a real match
|
||||
const KIND_BOOST = 0.02; // docs answer "how do I" better than a script header does
|
||||
|
||||
function detectIntent(q) {
|
||||
const hits = [];
|
||||
for (const i of INTENTS) if (i.re.test(q)) hits.push(i.section);
|
||||
return hits;
|
||||
}
|
||||
|
||||
function blobToVec(buf) {
|
||||
const b = Buffer.from(buf);
|
||||
return new Float32Array(b.buffer, b.byteOffset, b.length / 4);
|
||||
}
|
||||
|
||||
function dot(a, b) {
|
||||
let s = 0;
|
||||
for (let i = 0; i < a.length; i++) s += a[i] * b[i];
|
||||
return s;
|
||||
}
|
||||
|
||||
async function embedQuery(url, model, text, timeoutMs = 60000) {
|
||||
const res = await fetch(`${url.replace(/\/$/, '')}/api/embed`, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ model, input: text }),
|
||||
signal: AbortSignal.timeout(timeoutMs),
|
||||
});
|
||||
if (!res.ok) throw new Error(`embed HTTP ${res.status}`);
|
||||
const j = await res.json();
|
||||
if (!j.embeddings || !j.embeddings[0]) throw new Error('embed returned no vector');
|
||||
return Float32Array.from(j.embeddings[0]);
|
||||
}
|
||||
|
||||
// Keep at most `perFile` chunks from any one file, so a single large document cannot fill the
|
||||
// entire context window and crowd out a better answer living somewhere else.
|
||||
function diversify(rows, k, perFile) {
|
||||
const seen = new Map();
|
||||
const out = [];
|
||||
for (const r of rows) {
|
||||
const n = seen.get(r.path) || 0;
|
||||
if (n >= perFile) continue;
|
||||
seen.set(r.path, n + 1);
|
||||
out.push(r);
|
||||
if (out.length >= k) break;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function search(opts) {
|
||||
const {
|
||||
dbPath, url, embedModel, query,
|
||||
k = 8, perFile = 3, section = null, kind = null, minScore = 0.0,
|
||||
} = opts;
|
||||
|
||||
const db = new DatabaseSync(dbPath, { readOnly: true });
|
||||
|
||||
let sql = 'SELECT id,path,kind,section,heading,part,content,vector FROM vv_chunks';
|
||||
const where = [], args = [];
|
||||
if (section) { where.push('section = ?'); args.push(section); }
|
||||
if (kind) { where.push('kind = ?'); args.push(kind); }
|
||||
if (where.length) sql += ' WHERE ' + where.join(' AND ');
|
||||
|
||||
const rows = db.prepare(sql).all(...args);
|
||||
if (!rows.length) { db.close(); return { results: [], intents: [], scanned: 0 }; }
|
||||
|
||||
const qv = await embedQuery(url, embedModel, query);
|
||||
const intents = section ? [] : detectIntent(query);
|
||||
const wantDoc = /\b(how do i|steps|procedure|setup|install|troubleshoot|guide)\b/i.test(query);
|
||||
|
||||
const scored = rows.map(r => {
|
||||
let s = dot(qv, blobToVec(r.vector));
|
||||
if (intents.includes(r.section)) s += SECTION_BOOST;
|
||||
if (wantDoc && (r.kind === 'manual' || r.kind === 'readme')) s += KIND_BOOST;
|
||||
return {
|
||||
id: r.id, path: r.path, kind: r.kind, section: r.section,
|
||||
heading: r.heading, part: r.part, content: r.content, score: s,
|
||||
};
|
||||
});
|
||||
|
||||
scored.sort((a, b) => b.score - a.score);
|
||||
const kept = diversify(scored.filter(r => r.score >= minScore), k, perFile);
|
||||
db.close();
|
||||
return { results: kept, intents, scanned: rows.length };
|
||||
}
|
||||
|
||||
module.exports = { search, detectIntent, blobToVec, dot, embedQuery, INTENTS };
|
||||
@@ -569,6 +569,20 @@
|
||||
HOSTN_LLDAP_USER="admin" # lldap admin username
|
||||
HOSTN_LLDAP_PASS="" # lldap admin password
|
||||
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Ollama / AI ───────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Per-host because only some nodes actually have a GPU. A node with an empty OLLAMA_URL is not
|
||||
# an error — it falls through to the resolver and uses another node's Ollama over Tailscale.
|
||||
|
||||
# ━━━ Ollama ━━━
|
||||
HOSTN_OLLAMA_URL="" # e.g. http://localhost:11434 — empty if no local Ollama
|
||||
HOSTN_OLLAMA_CONTAINER="Ollama" # for watchdog / restart lists
|
||||
HOSTN_OLLAMA_GPU_UUID="" # pins Ollama to one card on multi-GPU hosts
|
||||
HOSTN_OLLAMA_MODEL="qwen2.5-coder:14b" # generation
|
||||
HOSTN_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — the generation model cannot embed
|
||||
|
||||
# ━━━ Authelia ━━━
|
||||
HOSTN_AUTHELIA_CONFIG="/mnt/user/appdata-Fallback/Critical-Data/Authelia/configuration.yml"
|
||||
HOSTN_AUTHELIA_CONTAINER="Authelia"
|
||||
|
||||
@@ -1593,6 +1593,54 @@
|
||||
SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity better than crashing mid-check
|
||||
SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting move better than crashing mid-move
|
||||
|
||||
# ==============================================================================================
|
||||
# ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
|
||||
# script to function — every feature that can lean on AI has a complete non-AI path, and the
|
||||
# AI path is an enhancement layered on top. If Ollama is unreachable, callers proceed without it.
|
||||
#
|
||||
# AI_ENABLED is necessary but not sufficient. Every feature stays individually off until it has
|
||||
# earned it — narration for months before anything is allowed near a decision.
|
||||
|
||||
# ━━━ AI Master Switch ━━━
|
||||
# Fail-closed: anything other than the literal "true" means off.
|
||||
AI_ENABLED=false
|
||||
AI_CONNECT_TIMEOUT=5 # seconds — probe when resolving which node has Ollama
|
||||
AI_REQUEST_TIMEOUT=240 # seconds — must clear a cold model load
|
||||
AI_RESOLVE_CACHE_TTL=300 # seconds — don't re-probe the mesh every invocation
|
||||
AI_MAX_RETRIES=1 # AI is enhancement; do not retry hard
|
||||
|
||||
# ━━━ AI Retrieval Index ━━━
|
||||
# The RAG index over this repo's own headers and documentation. Regenerable in minutes and
|
||||
# gitignored — it is derived data, never a source of truth.
|
||||
#
|
||||
# Only git-tracked files are ever indexed. Configurations/, State_Files/ and data/ are
|
||||
# gitignored, which is what makes it structurally impossible for a credential to reach the
|
||||
# index: the files holding them were never in the repo. Do not "improve" this to a filesystem
|
||||
# walk — an embedded secret cannot be rotated out of a vector.
|
||||
AI_INDEX_DB="$DATA_DIR/ai_index.db"
|
||||
AI_INDEX_BATCH=32 # chunks per embed request
|
||||
AI_INDEX_ON_PULL=false # re-index after a successful git pull once AI is in use
|
||||
AI_SEARCH_K=8 # chunks retrieved per query
|
||||
AI_SEARCH_PER_FILE=3 # cap per file so one document cannot fill the context
|
||||
|
||||
# ━━━ AI Feature Toggles ━━━
|
||||
# Tier 1 is narration — it cannot change a decision. Tier 2 adds context to a decision a script
|
||||
# already made. Tier 3 assists a human. Enable in that order, and give each one weeks.
|
||||
AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration
|
||||
AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition
|
||||
AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgement calls
|
||||
AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage
|
||||
AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance
|
||||
|
||||
# ━━━ AI Conf Write Access ━━━
|
||||
# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
|
||||
# never credentials, never a container name. An empty whitelist means no writes regardless of
|
||||
# the toggle.
|
||||
AI_CONF_WRITE_ENABLED=false
|
||||
AI_CONF_WRITE_KEYS=()
|
||||
|
||||
# ==============================================================================================
|
||||
# ──────────────────────── End Of User Variables ───────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
@@ -91,13 +91,13 @@
|
||||
# FALLBACK_TEST_BLOCK_WAIT
|
||||
# Seconds to wait in Phase 3 for fallback.sh to detect the outage.
|
||||
# Must be > FALLBACK_CHECK_INTERVAL + buffer. At 30s interval: use ≥60s.
|
||||
# (default: 60)
|
||||
# (shipped default: 150)
|
||||
#
|
||||
# FALLBACK_TEST_HANDBACK_WAIT
|
||||
# Seconds to wait in Phase 6 for fallback.sh to complete handback.
|
||||
# Must cover: FALLBACK_HANDBACK_STRIKES × FALLBACK_CHECK_INTERVAL + rsync
|
||||
# duration + container start time. At 3 strikes × 30s + ~2min rsync +
|
||||
# ~1min container start: use ≥240s. (default: 300)
|
||||
# ~1min container start: use ≥240s. (shipped default: 360)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
|
||||
+69
-2
@@ -1,7 +1,13 @@
|
||||
# Varaverk AI Integration — Design Notes
|
||||
|
||||
**Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no
|
||||
`AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives.
|
||||
**Status: retrieval is built; integration is not.** As of 2026-08-02 the `AI_*` and
|
||||
`HOST*_OLLAMA_*` variables exist in both confs and both templates, and `AI/` holds a working
|
||||
index and query path — see the RAG section at the end of this document and `AI/README-AI.md`.
|
||||
|
||||
Everything else below remains design only. **No Varaverk script consults AI.** Every
|
||||
`AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false with an empty whitelist, and
|
||||
host resolution across the mesh is specified but not implemented. Originally captured
|
||||
2026-08-01 so the reasoning survives.
|
||||
|
||||
Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for
|
||||
generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See
|
||||
@@ -490,3 +496,64 @@ new — the machinery already exists and was audited this session.
|
||||
possibly credentials pasted by a user.
|
||||
- Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the
|
||||
serialisation? Defer until an actual problem is felt.
|
||||
|
||||
---
|
||||
|
||||
## RAG — built 2026-08-02
|
||||
|
||||
Retrieval is live. `AI/` holds the implementation; `AI/README-AI.md` documents it in full. What
|
||||
follows is only what changed relative to the plan recorded above.
|
||||
|
||||
**Corpus is larger than estimated.** ~2,950 chunks across ~180 files, not the 690 header chunks
|
||||
projected. Sub-chunking is why — see below.
|
||||
|
||||
**Named-paragraph sub-chunking was necessary, and was not in the plan.** Section-level chunks
|
||||
alone were too coarse. `rsync.sh` documents fourteen safeguards in one 2.8k-char
|
||||
`OPERATIONAL SAFEGUARDS` block; a query about one of them scored 0.558, below unrelated chunks,
|
||||
because the other thirteen dominated the vector. Splitting on the named-paragraph titles the
|
||||
header convention already uses took the same query to 0.718 and first place. The parent section
|
||||
name is carried onto each sub-chunk so routing still works.
|
||||
|
||||
**Two chunker bugs worth remembering.** The last section in a header (RUNTIME MODES in bash,
|
||||
DEPENDS ON in a page) ran to EOF and swept up every unrelated comment in the file —
|
||||
`scheduler.php` alone produced an 11k-char chunk of unrelated inline comments. And title
|
||||
detection must require the *next* line to be indented; without that, any wrapped prose line
|
||||
became a spurious chunk boundary mid-sentence.
|
||||
|
||||
**Section routing is a boost, not a filter.** Intent detection is a heuristic and must not be
|
||||
able to exclude the chunk holding the answer. `--section=` forces a hard filter when wanted.
|
||||
|
||||
**Vectors arrive pre-normalised.** `nomic-embed-text` returns L2-normalised vectors (measured
|
||||
norm 1.0000001), so cosine is a plain dot product. No normalising step, no magnitude cache.
|
||||
|
||||
**`node:sqlite` over a native module.** Still flagged experimental, chosen because it needs no
|
||||
native compilation on Unraid. Acceptable because the index is disposable — if a Node upgrade
|
||||
breaks it, rebuild takes minutes. PHP reads the same float32 blobs with `unpack('f*', $blob)`
|
||||
when the UI needs them.
|
||||
|
||||
**Retrieval quality, measured.** 9/10 top-3 hit rate on known-answer questions; the tenth had
|
||||
the answer at ranks 2 and 3, so 10/10 for answer-present-in-context at k=6. Full retrieval plus
|
||||
generation runs about 43s warm.
|
||||
|
||||
### The finding that validated the whole thing
|
||||
|
||||
First real end-to-end question asked which variable controls the mover's grace window. The model
|
||||
answered `MOVER_STOP_TIMEOUT`, "defaults to 30 seconds", citing `mover_stop.sh › CONFIGURATION`.
|
||||
Variable correct; the 30 was wrong — the real value is 300. The model was quoting the header
|
||||
verbatim. **The header was stale.**
|
||||
|
||||
A sweep for the same pattern found six stale `(default: N)` claims across the repo — all
|
||||
corrected in the same pass. This is the operating principle for the folder:
|
||||
|
||||
> Retrieval is exactly as accurate as the documentation it points at. When an answer looks
|
||||
> wrong, check the cited source before blaming the model.
|
||||
|
||||
It also means the index is a documentation-drift detector, not only a question-answering tool.
|
||||
|
||||
### Still deliberately not built
|
||||
|
||||
Nothing consults this. Every `AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false
|
||||
with an empty key whitelist, and no watchdog, cleanup or fallback path calls it. Host resolution
|
||||
across the Tailscale mesh is specified above but not implemented — `ai_index.sh` and
|
||||
`ai_query.sh` currently require a local `HOST*_OLLAMA_URL` and fail with a clear message when it
|
||||
is empty, rather than silently probing the mesh.
|
||||
|
||||
@@ -74,7 +74,7 @@
|
||||
# master.conf
|
||||
#
|
||||
# MOVER_STOP_TIMEOUT
|
||||
# Seconds between wall warning and SIGTERM. (default: 30)
|
||||
# Seconds between wall warning and SIGTERM. (shipped default: 300)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
|
||||
+2
-2
@@ -161,14 +161,14 @@
|
||||
# Default retry attempts on rsync failure. (default: 3)
|
||||
#
|
||||
# SLEEP
|
||||
# Default seconds between retry attempts. (default: 60)
|
||||
# Default seconds between retry attempts. (shipped default: 300)
|
||||
#
|
||||
# RSYNC_MAX_RUNTIME_HOURS
|
||||
# Max hours a single transfer attempt may run before it's terminated and paused
|
||||
# for the next scheduled run. Protects the per-profile lock from being held
|
||||
# indefinitely by one huge/stuck transfer, starving other profiles of a turn.
|
||||
# Safe because DEFAULT_RSYNC_OPTS includes --partial — a paused transfer resumes
|
||||
# from where it left off, not from scratch. (default: 23)
|
||||
# from where it left off, not from scratch. (code fallback 23; shipped conf sets 19)
|
||||
#
|
||||
# ROOTFS_WARN
|
||||
# Abort threshold for remote rootfs percentage full. (default: 75)
|
||||
|
||||
@@ -85,7 +85,7 @@
|
||||
# master.conf
|
||||
#
|
||||
# REBOOT_SLEEP
|
||||
# Seconds between warning and shutdown sequence start. (default: 30)
|
||||
# Seconds between warning and shutdown sequence start. (shipped default: 300)
|
||||
#
|
||||
# REBOOT_VM_WAIT
|
||||
# Seconds to wait for VMs to shut down gracefully. (default: 30)
|
||||
|
||||
Reference in New Issue
Block a user