From 6a959fb5e4a3658ddf1312bf3dd6bfbb895102d2 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sun, 2 Aug 2026 01:21:24 -0400 Subject: [PATCH] Add AI entry points, conf schema, and folder docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate, root check, lock, dry-run, status — with Node doing only the vector maths and SQLite blobs, the same split api_cache_writer.sh uses for PHP. AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass. Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes disabled with an empty whitelist. Nothing in the ecosystem consults it. --- AI/README-AI.md | 217 ++++++++++++++++++++++++++++ AI/ai_index.sh | 244 ++++++++++++++++++++++++++++++++ AI/ai_query.sh | 184 ++++++++++++++++++++++++ AI/lib/index.js | 16 ++- Deployment/host.conf.template | 14 ++ Deployment/master.conf.template | 48 +++++++ Notes_AI-Design.md | 71 +++++++++- 7 files changed, 785 insertions(+), 9 deletions(-) create mode 100644 AI/README-AI.md create mode 100755 AI/ai_index.sh create mode 100755 AI/ai_query.sh diff --git a/AI/README-AI.md b/AI/README-AI.md new file mode 100644 index 0000000..480cc44 --- /dev/null +++ b/AI/README-AI.md @@ -0,0 +1,217 @@ +# ━━━━━ AI ━━━━━ + +Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an +answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources. + +Everything here is **off by default and optional**. Varaverk works exactly as well with +`AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder. + +| File | Role | +|------|------| +| `ai_index.sh` | Build / refresh the retrieval index | +| `ai_query.sh` | Ask a question, or search the index directly | +| `lib/chunk.js` | Split repo files into retrieval units | +| `lib/index.js` | Embed chunks, store vectors in SQLite | +| `lib/search.js` | Embed a query, score it, rank results | +| `lib/cli.js` | Argument bridge the bash entry points call | + +--- + +## ━━━ WHY THIS WORKS AT ALL ━━━ + +**Because the header audit came first.** + +The single worst failure in naive retrieval is a chunk that contains half of one idea and half +of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one +vector. That problem does not exist here, because every script in the repo carries the same six +sections at exact, greppable boundaries: + +``` +PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS + → CONFIGURATION → RUNTIME MODES +``` + +Split on those and every chunk is a coherent unit by construction. No token windows, no overlap +heuristics, no tuning. + +**And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it +does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that +`removeCompletedDownloads` being true on both arrs is intentional. Those live in +`DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth +more than an equivalent pile of source. + +--- + +## ━━━ SECTION ROUTING ━━━ + +Every chunk stores its section name as its own column, so a question's *shape* can steer +retrieval before similarity is even considered: + +| Question shape | Steered toward | +|----------------|----------------| +| "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` | +| "which variable controls X" | `CONFIGURATION` | +| "does this take --dry-run" | `RUNTIME MODES` | +| "why is it built this way" | `DESIGN PRINCIPLES` | +| "how does X work" | `OPERATIONAL MODEL` | + +Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic +must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a +hard filter when you actually want one. + +--- + +## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━ + +Sections alone were not granular enough, and the failure was instructive. + +`rsync.sh` documents fourteen distinct safeguards in one 2.8k-character +`OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the +correct chunk scored **0.558** — below unrelated chunks from other files — because the other +thirteen safeguards dominated the vector. + +The header convention writes each safeguard as a named paragraph: an unindented title, an +indented body. Splitting on those titles took the same query to **0.718** and first place. + +``` +Merge-Run Delete Interlock ← its own chunk + --delete is applied only when pass 1 completed... +``` + +The parent section name is carried onto every sub-chunk, so routing still works. The title +detection requires the *next* line to be indented — without that check, any wrapped prose line +became a spurious boundary mid-sentence. + +--- + +## ━━━ WHAT IS INDEXED ━━━ + +Roughly 2,900 chunks across ~180 files: + +| Kind | Source | +|------|--------| +| `header` | Script headers — bash and PHP, sections and named paragraphs | +| `readme` | Every `README-*.md` | +| `manual` | Every `Manual-*.md` | +| `template` | `Deployment/*.template` — the versioned conf schema | +| `doc` | Top-level `README.md`, `Manual.md`, design notes | + +**Script bodies are not indexed.** Headers state intent, code states mechanism; for the +questions this answers, intent retrieves better and costs far less. + +--- + +## ━━━ THE SAFETY BOUNDARY ━━━ + +**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model. + +`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential +was never in the repo to begin with. The index therefore describes the full conf schema (via the +tracked templates, which carry all the explanatory comments) while structurally **never +containing a secret**. + +> Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret +> averaged into a 768-dimension float cannot be found, let alone removed. + +A live conf value the model genuinely needs should arrive through a tool call at query time, +subject to redaction — never baked into a vector at index time. + +--- + +## ━━━ HOW IT IS BUILT ━━━ + +**SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds +per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and +back up, in exchange for nothing at this scale. + +**Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised +vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP +reads the same blobs with `unpack('f*', $blob)` when the UI needs them. + +**Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A +no-op refresh takes about 70 ms; a full rebuild takes a few minutes. + +**Node for the maths, bash for everything else.** The bash entry points own configuration, +gating, locking and logging exactly as every other Varaverk job does. Node owns only float +vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP. + +> `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it +> needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is +> regenerable in minutes — this is a disposable artefact, not a datastore. + +--- + +## ━━━ USAGE ━━━ + +```bash +# Build (needs AI_ENABLED=true) +bash AI/ai_index.sh # incremental +bash AI/ai_index.sh --force # full rebuild +bash AI/ai_index.sh --dry-run # what would be indexed; contacts nothing +bash AI/ai_index.sh --status # size, counts, last build + +# Ask +bash AI/ai_query.sh "what stops rsync and the mover running at once" +bash AI/ai_query.sh --search "why are the cache writers lockless" +bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period" +bash AI/ai_query.sh --json "..." # for other scripts +``` + +**`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated. +When the answer matters, use it — or read the sources the generated answer cites. + +--- + +## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━ + +The generation prompt instructs the model to answer only from retrieved context and to say what +is missing rather than fill the gap from general knowledge. That instruction matters here more +than usual: this repo's conventions are frequently *not* the conventional ones, and a confident +generic answer about rsync, Docker or systemd is worse than no answer. + +It works — and its faithfulness cut both ways on the first real test. Asked which variable +controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30 +seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the +real value is 300 — and the model was quoting the header verbatim. **The header was stale.** + +That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The +lesson is the operating principle for this whole folder: + +> Retrieval is exactly as accurate as the documentation it points at. When an answer looks +> wrong, check the cited source before blaming the model — it is usually reporting a real +> problem in the repo. + +--- + +## ━━━ WHAT THIS DOES NOT DO ━━━ + +- **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with + an empty key whitelist. Nothing in this folder writes a setting. +- **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The + per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately. +- **It does not index code.** Headers and docs only. +- **It does not sync.** The index is host-local and gitignored. Each node builds its own. +- **It is not required.** Every script runs identically with `AI_ENABLED=false`. + +See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the +Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be +built before any of that is enabled. + +--- + +## ━━━ SCHEDULING ━━━ + +Not scheduled by default. When you want the index to track the repo automatically, the natural +home is after a successful pull — the only moment the corpus actually changes: + +```bash +# master.conf → DAILY_MAINTENANCE_SCRIPTS, after git_pull_execute.sh +"AI/ai_index.sh" +``` + +`AI_INDEX_ON_PULL` exists in `master.conf` as the intended gate for wiring this into +`git_pull_execute.sh` directly. It is not yet consumed by anything — the variable is reserved, +not live. + +An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing +and a pull that changed twelve files costs a few seconds. diff --git a/AI/ai_index.sh b/AI/ai_index.sh new file mode 100755 index 0000000..9f9db5d --- /dev/null +++ b/AI/ai_index.sh @@ -0,0 +1,244 @@ +#!/bin/bash +# ============================================================================================== +# ━━━ AI Retrieval Index Builder ━━━ +# ============================================================================================== +# +# PURPOSE +# ============================================================================================== +# Builds and refreshes the retrieval index over Varaverk's own documentation — every script +# header, folder README and Manual, and the conf templates. Chunks each file on the section +# boundaries the header convention already defines, embeds each chunk through Ollama, and +# stores the vectors in SQLite for AI/ai_query.sh to search. +# +# The index is derived data. It is gitignored, host-local, and rebuildable from the repo in +# minutes — nothing depends on it surviving. +# +# ============================================================================================== +# OPERATIONAL MODEL +# ============================================================================================== +# Chunking mirrors the header convention rather than using a fixed token window: +# +# bash headers split on the six section names, then sub-split named-paragraph +# safeguards and principles so one question finds one answer +# PHP headers same six names plus the per-layer tails (EXPORTS, RENDERS, ...) +# markdown split on headings +# conf templates split on the ━━━ section rules +# +# Every chunk keeps its section name as a field, which is what lets a query about a safeguard +# be steered toward OPERATIONAL SAFEGUARDS chunks before similarity is considered. +# +# Incremental by file mtime. A file whose mtime has not moved is skipped without being read, +# so a routine refresh costs well under a second and a full rebuild costs a few minutes. +# +# Heavy lifting runs in Node — float vector maths and SQLite BLOBs are genuinely awkward in +# bash. This follows the api_cache_writer.sh precedent: a bash shim owning config, gating, +# locking and logging, in front of the language that fits the work. +# +# ============================================================================================== +# DESIGN PRINCIPLES +# ============================================================================================== +# +# Tracked Files Only +# Indexes exactly what `git ls-files` reports. Configurations/, State_Files/ and data/ are +# gitignored, so it is structurally impossible for a credential to reach the index — the +# files holding them were never in the repo. This must never become a filesystem walk: a +# secret written into a vector cannot be rotated back out of it. +# +# The Index Is Disposable +# Stored under DATA_DIR, gitignored, and never synced to a partner. Losing it costs one +# rebuild. Nothing reads it as a source of truth — it points at files, and the files are +# the truth. +# +# Documentation Is The Corpus, Not The Code +# Script bodies are not indexed. The headers state intent and the code states mechanism; +# for the questions this answers, intent retrieves far better and is far cheaper. +# +# Off By Default +# Does nothing unless AI_ENABLED is true. A node with AI off never pays for this. +# +# ============================================================================================== +# OPERATIONAL SAFEGUARDS +# ============================================================================================== +# +# Fail-Closed Gate +# Exits cleanly unless AI_ENABLED is exactly "true". Any other value, including unset, +# means off. +# +# Root Enforcement +# Writes into DATA_DIR alongside other Varaverk state. +# +# Concurrency Lock +# acquire_lock() prevents two indexers racing on the same database. +# +# Dependency Verification +# Verifies node and the Ollama endpoint before touching the database. A missing dependency +# is reported and exits non-zero rather than leaving a half-built index. +# +# Reachability Pre-flight +# Probes the resolved Ollama URL with AI_CONNECT_TIMEOUT before starting. An unreachable +# endpoint aborts immediately instead of failing once per batch across the whole corpus. +# +# Partial Failure Is Not Recorded As Success +# A file whose chunks all failed to embed keeps its previous rows and its old mtime, so the +# next run retries it. A run with any failed batch exits 3. +# +# Atomic Per-Run Write +# All database changes commit in one transaction. An interrupted run leaves the previous +# index intact rather than a partially rewritten one. +# +# Deleted Files Are Removed From The Index +# A file that has left the repo has its chunks deleted, so retrieval cannot cite something +# that no longer exists. +# +# ============================================================================================== +# CONFIGURATION +# ============================================================================================== +# +# master.conf +# +# AI_ENABLED +# Master switch. Fail-closed — must be exactly "true". +# +# AI_INDEX_DB +# SQLite index path. (shipped default: $DATA_DIR/ai_index.db) +# +# AI_INDEX_BATCH +# Chunks per embed request. (shipped default: 32) +# +# AI_CONNECT_TIMEOUT +# Seconds for the reachability probe. (shipped default: 5) +# +# AI_REQUEST_TIMEOUT +# Seconds for a single embed batch. (shipped default: 240) +# +# host*.conf +# +# HOST*_OLLAMA_URL +# This host's Ollama endpoint. Empty means no local Ollama. +# +# HOST*_OLLAMA_EMBED_MODEL +# Embedding model. The generation model cannot embed. +# +# ============================================================================================== +# RUNTIME MODES +# ============================================================================================== +# +# ai_index.sh +# Incremental refresh — only files whose mtime moved are re-embedded. +# +# ai_index.sh --force +# Full rebuild. Discards the existing index and re-embeds every chunk. +# +# ai_index.sh --dry-run +# Report what would be indexed. Contacts nothing and writes nothing. +# +# ai_index.sh --status +# Show index location, size, chunk counts by kind and section, and last build time. +# +# ai_index.sh --log +# Verbose — per-batch embedding progress. +# +# ============================================================================================== + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/../load_config.sh" + +detect_hosts + +DRY_RUN=false; FORCE=false; STATUS=false; LOG=false +for arg in "$@"; do + case "$arg" in + --dry-run) DRY_RUN=true ;; + --force) FORCE=true ;; + --status) STATUS=true ;; + --log) LOG=true ;; + *) echo "Unknown option: $arg" >&2; exit 1 ;; + esac +done + +CLI="${SCRIPT_DIR}/lib/cli.js" +DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}" + +_url_var="${MY_ID}_OLLAMA_URL" +_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL" +OLLAMA_URL="${!_url_var:-}" +EMBED_MODEL="${!_emb_var:-nomic-embed-text}" + +# ── Status ──────────────────────────────────────────────────────────────────────────────────── +if [[ "$STATUS" == true ]]; then + echo "$ICON_GEAR AI Index Status" + echo " Enabled: ${AI_ENABLED:-false}" + echo " Database: $DB" + if [[ -f "$DB" ]]; then + echo " Size: $(du -h "$DB" 2>/dev/null | cut -f1)" + echo " Chunks: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null || echo '?')" + echo " Files: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_files;' 2>/dev/null || echo '?')" + _last=$(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='last_index';" 2>/dev/null) + [[ -n "$_last" ]] && echo " Last built: $(date -d "@$_last" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)" + echo " Model: $(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='embed_model';" 2>/dev/null || echo '?')" + echo " By kind:" + sqlite3 "$DB" "SELECT ' '||kind||': '||COUNT(*) FROM vv_chunks GROUP BY kind ORDER BY COUNT(*) DESC;" 2>/dev/null + else + echo " Database: not built yet" + fi + echo " Ollama: ${OLLAMA_URL:-}" + echo " Embed: $EMBED_MODEL" + exit 0 +fi + +# ── Gate ────────────────────────────────────────────────────────────────────────────────────── +if [[ "${AI_ENABLED:-false}" != "true" ]]; then + log "AI_ENABLED is not true — skipping index build" + exit 0 +fi + +if [[ "$DRY_RUN" == false && "$EUID" -ne 0 ]]; then + error "Must be run as root" + exit 1 +fi + +acquire_lock + +command -v node >/dev/null 2>&1 || { error "node not found — required to build the index"; exit 1; } +[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; } + +if [[ -z "$OLLAMA_URL" ]]; then + error "${MY_ID}_OLLAMA_URL is empty — no local Ollama to index against" + exit 1 +fi + +# Pre-flight: fail once, up front, rather than once per batch across the whole corpus. +if [[ "$DRY_RUN" == false ]]; then + if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then + error "Ollama unreachable at $OLLAMA_URL" + exit 1 + fi +fi + +# ── Build ───────────────────────────────────────────────────────────────────────────────────── +_args=( + index + "--root=${SCRIPTS_DIR}" + "--db=${DB}" + "--url=${OLLAMA_URL}" + "--model=${EMBED_MODEL}" + "--batch=${AI_INDEX_BATCH:-32}" + "--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))" +) +[[ "$FORCE" == true ]] && _args+=(--force) +[[ "$DRY_RUN" == true ]] && _args+=(--dry-run) +[[ "$LOG" == false ]] && _args+=(--quiet) + +log "$ICON_GEAR Building AI index → $DB" +node --no-warnings "$CLI" "${_args[@]}" +_rc=$? + +case "$_rc" in + 0) log "$ICON_DONE AI index build complete" ;; + 3) warn "AI index built with some batches failed — those files will retry next run" ;; + *) error "AI index build failed (exit $_rc)" ;; +esac + +exit "$_rc" diff --git a/AI/ai_query.sh b/AI/ai_query.sh new file mode 100755 index 0000000..4dfb483 --- /dev/null +++ b/AI/ai_query.sh @@ -0,0 +1,184 @@ +#!/bin/bash +# ============================================================================================== +# ━━━ AI Retrieval Query ━━━ +# ============================================================================================== +# +# PURPOSE +# ============================================================================================== +# Answers questions about Varaverk from Varaverk's own documentation. Embeds the question, +# retrieves the closest chunks from the index AI/ai_index.sh built, and either prints them +# directly or passes them to the generation model as grounding context. +# +# ============================================================================================== +# OPERATIONAL MODEL +# ============================================================================================== +# Two modes over the same retrieval: +# +# --search print the matching chunks and their sources. No generation model involved, +# so it is fast and its output is verbatim repo text. +# (default) retrieve, then ask the generation model to answer strictly from what was +# retrieved, citing each claim. +# +# Retrieval is steered by question shape. A question about what prevents something is pushed +# toward OPERATIONAL SAFEGUARDS chunks, one about a variable toward CONFIGURATION, one asking +# why toward DESIGN PRINCIPLES. This is a score boost, not a filter — a heuristic must not be +# able to exclude the chunk that actually holds the answer. +# +# ============================================================================================== +# DESIGN PRINCIPLES +# ============================================================================================== +# +# Grounded Or Silent +# The prompt instructs the model to answer only from retrieved context and to say what is +# missing rather than fill the gap. This repo's conventions are frequently not the +# conventional ones, and a confident generic answer about rsync or Docker is worse here +# than no answer. +# +# Sources Are Always Shown +# Every answer prints the chunks it drew on. An answer that cannot be traced back to a file +# is not usable for changing anything. +# +# Search Is The Trustworthy Mode +# --search returns repo text with nothing generated. When an answer matters, use it. +# +# Read-Only +# Retrieves and answers. Nothing here writes conf, touches state, or runs another script. +# +# ============================================================================================== +# OPERATIONAL SAFEGUARDS +# ============================================================================================== +# +# Fail-Closed Gate +# Exits cleanly unless AI_ENABLED is exactly "true". +# +# No Root Required +# Reads the index and calls Ollama. Nothing it does needs privilege, so it does not ask for +# any — this is the one AI script an ordinary user should be able to run. +# +# Missing Index Is Reported, Not Built +# An absent index exits with guidance to run ai_index.sh. Building a corpus-wide index as a +# side effect of a question would turn a two-second query into a several-minute one. +# +# Reachability Pre-flight +# Probes Ollama with AI_CONNECT_TIMEOUT before embedding, so an unreachable endpoint fails +# immediately with a clear message. +# +# Bounded Generation +# The request is capped at AI_REQUEST_TIMEOUT. A wedged model cannot hang the caller. +# +# ============================================================================================== +# CONFIGURATION +# ============================================================================================== +# +# master.conf +# +# AI_ENABLED +# Master switch. Fail-closed — must be exactly "true". +# +# AI_INDEX_DB +# SQLite index to search. (shipped default: $DATA_DIR/ai_index.db) +# +# AI_SEARCH_K +# Chunks retrieved per query. (shipped default: 8) +# +# AI_SEARCH_PER_FILE +# Cap per file, so one document cannot fill the context. (shipped default: 3) +# +# AI_REQUEST_TIMEOUT +# Seconds allowed for generation. (shipped default: 240) +# +# host*.conf +# +# HOST*_OLLAMA_URL / HOST*_OLLAMA_MODEL / HOST*_OLLAMA_EMBED_MODEL +# +# ============================================================================================== +# RUNTIME MODES +# ============================================================================================== +# +# ai_query.sh "your question" +# Retrieve and answer, with sources. +# +# ai_query.sh --search "your question" +# Print matching chunks only. No generation. +# +# ai_query.sh --section=CONFIGURATION "your question" +# Restrict retrieval to one header section. +# +# ai_query.sh --json "your question" +# Machine-readable output for other scripts. +# +# ai_query.sh --status +# Show index and endpoint state. +# +# ============================================================================================== + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "${SCRIPT_DIR}/../load_config.sh" + +detect_hosts + +SEARCH_ONLY=false; JSON=false; STATUS=false; SECTION=""; QUERY="" +for arg in "$@"; do + case "$arg" in + --search) SEARCH_ONLY=true ;; + --json) JSON=true ;; + --status) STATUS=true ;; + --section=*) SECTION="${arg#*=}" ;; + --*) echo "Unknown option: $arg" >&2; exit 1 ;; + *) QUERY="$arg" ;; + esac +done + +CLI="${SCRIPT_DIR}/lib/cli.js" +DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}" + +_url_var="${MY_ID}_OLLAMA_URL" +_gen_var="${MY_ID}_OLLAMA_MODEL" +_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL" +OLLAMA_URL="${!_url_var:-}" +GEN_MODEL="${!_gen_var:-}" +EMBED_MODEL="${!_emb_var:-nomic-embed-text}" + +if [[ "$STATUS" == true ]]; then + echo "$ICON_GEAR AI Query Status" + echo " Enabled: ${AI_ENABLED:-false}" + echo " Index: $DB $([[ -f "$DB" ]] && echo "($(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null) chunks)" || echo '(not built)')" + echo " Ollama: ${OLLAMA_URL:-}" + echo " Generate: ${GEN_MODEL:-}" + echo " Embed: $EMBED_MODEL" + exit 0 +fi + +if [[ "${AI_ENABLED:-false}" != "true" ]]; then + echo "AI_ENABLED is not true — AI features are off" >&2 + exit 0 +fi + +[[ -z "$QUERY" ]] && { echo "usage: ai_query.sh [--search] [--section=NAME] \"your question\"" >&2; exit 1; } +[[ -f "$DB" ]] || { error "No index at $DB — run AI/ai_index.sh first"; exit 1; } +[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; } +command -v node >/dev/null 2>&1 || { error "node not found"; exit 1; } +[[ -z "$OLLAMA_URL" ]] && { error "${MY_ID}_OLLAMA_URL is empty"; exit 1; } + +if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then + error "Ollama unreachable at $OLLAMA_URL" + exit 1 +fi + +_args=("--db=${DB}" "--url=${OLLAMA_URL}" "--query=${QUERY}") +[[ -n "$SECTION" ]] && _args+=("--section=${SECTION}") +[[ "$JSON" == true ]] && _args+=(--json) + +if [[ "$SEARCH_ONLY" == true ]]; then + node --no-warnings "$CLI" search "${_args[@]}" \ + "--model=${EMBED_MODEL}" \ + "--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}" +else + [[ -z "$GEN_MODEL" ]] && { error "${MY_ID}_OLLAMA_MODEL is empty — needed for generation"; exit 1; } + node --no-warnings "$CLI" ask "${_args[@]}" \ + "--model=${GEN_MODEL}" "--embed-model=${EMBED_MODEL}" \ + "--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}" \ + "--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))" +fi diff --git a/AI/lib/index.js b/AI/lib/index.js index 158203c..56e64b8 100644 --- a/AI/lib/index.js +++ b/AI/lib/index.js @@ -122,13 +122,9 @@ async function buildIndex(opts) { let chunks = []; try { chunks = chunkFile(abs, rel); } catch (e) { continue; } - if (!chunks.length) { - if (db) { - db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(rel); - db.prepare('INSERT OR REPLACE INTO vv_files VALUES (?,?,?,?)').run(rel, mtime, 0, now); - } - continue; - } + // A file that yields no chunks still gets a vv_files row so it is not re-chunked every + // run. Queued rather than written here, so every database change lands in the single + // commit below — a run interrupted mid-embed must leave the index exactly as it was. pending.push({ rel, mtime, chunks }); } @@ -166,6 +162,12 @@ async function buildIndex(opts) { db.exec('BEGIN'); try { for (const f of pending) { + // Nothing to index in this file at all — record it so it is not re-chunked next run. + if (!f.chunks.length) { + db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel); + insF.run(f.rel, f.mtime, 0, now); + continue; + } const embedded = f.chunks.filter(c => c.__vec); // A file whose chunks all failed to embed keeps its previous rows and its old mtime, // so the next run retries it rather than recording a half-indexed file as current. diff --git a/Deployment/host.conf.template b/Deployment/host.conf.template index c829532..88f950e 100644 --- a/Deployment/host.conf.template +++ b/Deployment/host.conf.template @@ -569,6 +569,20 @@ HOSTN_LLDAP_USER="admin" # lldap admin username HOSTN_LLDAP_PASS="" # lldap admin password + +# ============================================================================================== +# ── Ollama / AI ─────────────────────────────────────────────────────────────────────────────── +# ============================================================================================== +# Per-host because only some nodes actually have a GPU. A node with an empty OLLAMA_URL is not +# an error — it falls through to the resolver and uses another node's Ollama over Tailscale. + +# ━━━ Ollama ━━━ + HOSTN_OLLAMA_URL="" # e.g. http://localhost:11434 — empty if no local Ollama + HOSTN_OLLAMA_CONTAINER="Ollama" # for watchdog / restart lists + HOSTN_OLLAMA_GPU_UUID="" # pins Ollama to one card on multi-GPU hosts + HOSTN_OLLAMA_MODEL="qwen2.5-coder:14b" # generation + HOSTN_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — the generation model cannot embed + # ━━━ Authelia ━━━ HOSTN_AUTHELIA_CONFIG="/mnt/user/appdata-Fallback/Critical-Data/Authelia/configuration.yml" HOSTN_AUTHELIA_CONTAINER="Authelia" diff --git a/Deployment/master.conf.template b/Deployment/master.conf.template index be2f41d..af4596e 100644 --- a/Deployment/master.conf.template +++ b/Deployment/master.conf.template @@ -1593,6 +1593,54 @@ SYS_WATCHDOG_ABORT_ON_PARITY=false # aborting parity better than crashing mid-check SYS_WATCHDOG_ABORT_ON_MOVER=false # aborting move better than crashing mid-move +# ============================================================================================== +# ── AI / RAG ────────────────────────────────────────────────────────────────────────────────── +# ============================================================================================== +# Varaverk works exactly as well with AI off as with it on. Nothing below is required for any +# script to function — every feature that can lean on AI has a complete non-AI path, and the +# AI path is an enhancement layered on top. If Ollama is unreachable, callers proceed without it. +# +# AI_ENABLED is necessary but not sufficient. Every feature stays individually off until it has +# earned it — narration for months before anything is allowed near a decision. + +# ━━━ AI Master Switch ━━━ +# Fail-closed: anything other than the literal "true" means off. + AI_ENABLED=false + AI_CONNECT_TIMEOUT=5 # seconds — probe when resolving which node has Ollama + AI_REQUEST_TIMEOUT=240 # seconds — must clear a cold model load + AI_RESOLVE_CACHE_TTL=300 # seconds — don't re-probe the mesh every invocation + AI_MAX_RETRIES=1 # AI is enhancement; do not retry hard + +# ━━━ AI Retrieval Index ━━━ +# The RAG index over this repo's own headers and documentation. Regenerable in minutes and +# gitignored — it is derived data, never a source of truth. +# +# Only git-tracked files are ever indexed. Configurations/, State_Files/ and data/ are +# gitignored, which is what makes it structurally impossible for a credential to reach the +# index: the files holding them were never in the repo. Do not "improve" this to a filesystem +# walk — an embedded secret cannot be rotated out of a vector. + AI_INDEX_DB="$DATA_DIR/ai_index.db" + AI_INDEX_BATCH=32 # chunks per embed request + AI_INDEX_ON_PULL=false # re-index after a successful git pull once AI is in use + AI_SEARCH_K=8 # chunks retrieved per query + AI_SEARCH_PER_FILE=3 # cap per file so one document cannot fill the context + +# ━━━ AI Feature Toggles ━━━ +# Tier 1 is narration — it cannot change a decision. Tier 2 adds context to a decision a script +# already made. Tier 3 assists a human. Enable in that order, and give each one weeks. + AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration + AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition + AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgement calls + AI_ASSIST_CLEANUP=false # tier 2 — orphan and stuck-import triage + AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance + +# ━━━ AI Conf Write Access ━━━ +# Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths, +# never credentials, never a container name. An empty whitelist means no writes regardless of +# the toggle. + AI_CONF_WRITE_ENABLED=false + AI_CONF_WRITE_KEYS=() + # ============================================================================================== # ──────────────────────── End Of User Variables ─────────────────────────────────────────────── # ============================================================================================== diff --git a/Notes_AI-Design.md b/Notes_AI-Design.md index 834eae3..e885238 100644 --- a/Notes_AI-Design.md +++ b/Notes_AI-Design.md @@ -1,7 +1,13 @@ # Varaverk AI Integration — Design Notes -**Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no -`AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives. +**Status: retrieval is built; integration is not.** As of 2026-08-02 the `AI_*` and +`HOST*_OLLAMA_*` variables exist in both confs and both templates, and `AI/` holds a working +index and query path — see the RAG section at the end of this document and `AI/README-AI.md`. + +Everything else below remains design only. **No Varaverk script consults AI.** Every +`AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false with an empty whitelist, and +host resolution across the mesh is specified but not implemented. Originally captured +2026-08-01 so the reasoning survives. Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See @@ -490,3 +496,64 @@ new — the machinery already exists and was audited this session. possibly credentials pasted by a user. - Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the serialisation? Defer until an actual problem is felt. + +--- + +## RAG — built 2026-08-02 + +Retrieval is live. `AI/` holds the implementation; `AI/README-AI.md` documents it in full. What +follows is only what changed relative to the plan recorded above. + +**Corpus is larger than estimated.** ~2,950 chunks across ~180 files, not the 690 header chunks +projected. Sub-chunking is why — see below. + +**Named-paragraph sub-chunking was necessary, and was not in the plan.** Section-level chunks +alone were too coarse. `rsync.sh` documents fourteen safeguards in one 2.8k-char +`OPERATIONAL SAFEGUARDS` block; a query about one of them scored 0.558, below unrelated chunks, +because the other thirteen dominated the vector. Splitting on the named-paragraph titles the +header convention already uses took the same query to 0.718 and first place. The parent section +name is carried onto each sub-chunk so routing still works. + +**Two chunker bugs worth remembering.** The last section in a header (RUNTIME MODES in bash, +DEPENDS ON in a page) ran to EOF and swept up every unrelated comment in the file — +`scheduler.php` alone produced an 11k-char chunk of unrelated inline comments. And title +detection must require the *next* line to be indented; without that, any wrapped prose line +became a spurious chunk boundary mid-sentence. + +**Section routing is a boost, not a filter.** Intent detection is a heuristic and must not be +able to exclude the chunk holding the answer. `--section=` forces a hard filter when wanted. + +**Vectors arrive pre-normalised.** `nomic-embed-text` returns L2-normalised vectors (measured +norm 1.0000001), so cosine is a plain dot product. No normalising step, no magnitude cache. + +**`node:sqlite` over a native module.** Still flagged experimental, chosen because it needs no +native compilation on Unraid. Acceptable because the index is disposable — if a Node upgrade +breaks it, rebuild takes minutes. PHP reads the same float32 blobs with `unpack('f*', $blob)` +when the UI needs them. + +**Retrieval quality, measured.** 9/10 top-3 hit rate on known-answer questions; the tenth had +the answer at ranks 2 and 3, so 10/10 for answer-present-in-context at k=6. Full retrieval plus +generation runs about 43s warm. + +### The finding that validated the whole thing + +First real end-to-end question asked which variable controls the mover's grace window. The model +answered `MOVER_STOP_TIMEOUT`, "defaults to 30 seconds", citing `mover_stop.sh › CONFIGURATION`. +Variable correct; the 30 was wrong — the real value is 300. The model was quoting the header +verbatim. **The header was stale.** + +A sweep for the same pattern found six stale `(default: N)` claims across the repo — all +corrected in the same pass. This is the operating principle for the folder: + +> Retrieval is exactly as accurate as the documentation it points at. When an answer looks +> wrong, check the cited source before blaming the model. + +It also means the index is a documentation-drift detector, not only a question-answering tool. + +### Still deliberately not built + +Nothing consults this. Every `AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false +with an empty key whitelist, and no watchdog, cleanup or fallback path calls it. Host resolution +across the Tailscale mesh is specified above but not implemented — `ai_index.sh` and +`ai_query.sh` currently require a local `HOST*_OLLAMA_URL` and fail with a clear message when it +is empty, rather than silently probing the mesh.