Add AI entry points, conf schema, and folder docs

ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate, root check, lock, dry-run, status — with Node doing only the vector maths and SQLite blobs, the same split api_cache_writer.sh uses for PHP. AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass. Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes disabled with an empty whitelist. Nothing in the ecosystem consults it.
2026-08-02 01:21:24 -04:00
parent b00688bad1
commit 6a959fb5e4
7 changed files with 785 additions and 9 deletions
@@ -0,0 +1,217 @@
 # ━━━━━ AI ━━━━━
 Retrieval over Varaverk's own documentation. Ask the system a question about itself and get an
 answer grounded in its actual headers, READMEs, Manuals and conf templates — with sources.
 Everything here is **off by default and optional**. Varaverk works exactly as well with
 `AI_ENABLED=false` as with it true. Nothing in the ecosystem depends on this folder.
 | File | Role |
 |------|------|
 | `ai_index.sh` | Build / refresh the retrieval index |
 | `ai_query.sh` | Ask a question, or search the index directly |
 | `lib/chunk.js` | Split repo files into retrieval units |
 | `lib/index.js` | Embed chunks, store vectors in SQLite |
 | `lib/search.js` | Embed a query, score it, rank results |
 | `lib/cli.js` | Argument bridge the bash entry points call |
 ---
 ## ━━━ WHY THIS WORKS AT ALL ━━━
 **Because the header audit came first.**
 The single worst failure in naive retrieval is a chunk that contains half of one idea and half
 of another — a fixed-size window cutting mid-thought, embedding two unrelated things as one
 vector. That problem does not exist here, because every script in the repo carries the same six
 sections at exact, greppable boundaries:
 ```
 PURPOSE → OPERATIONAL MODEL → DESIGN PRINCIPLES → OPERATIONAL SAFEGUARDS
        → CONFIGURATION → RUNTIME MODES
 ```
 Split on those and every chunk is a coherent unit by construction. No token windows, no overlap
 heuristics, no tuning.
 **And because the headers say *why*.** A model can read `mover_stop.sh` and describe what it
 does. It cannot look at that code and know the cache writers are lockless *on purpose*, or that
 `removeCompletedDownloads` being true on both arrs is intentional. Those live in
 `DESIGN PRINCIPLES` and `OPERATIONAL SAFEGUARDS`, which is precisely what makes this index worth
 more than an equivalent pile of source.
 ---
 ## ━━━ SECTION ROUTING ━━━
 Every chunk stores its section name as its own column, so a question's *shape* can steer
 retrieval before similarity is even considered:
 | Question shape | Steered toward |
 |----------------|----------------|
 | "what stops X and Y overlapping" | `OPERATIONAL SAFEGUARDS` |
 | "which variable controls X" | `CONFIGURATION` |
 | "does this take --dry-run" | `RUNTIME MODES` |
 | "why is it built this way" | `DESIGN PRINCIPLES` |
 | "how does X work" | `OPERATIONAL MODEL` |
 Applied as a **score boost, not a filter**. Intent detection is a heuristic, and a heuristic
 must never be able to exclude the one chunk that holds the answer. `--section=NAME` forces a
 hard filter when you actually want one.
 ---
 ## ━━━ NAMED-PARAGRAPH SUB-CHUNKING ━━━
 Sections alone were not granular enough, and the failure was instructive.
 `rsync.sh` documents fourteen distinct safeguards in one 2.8k-character
 `OPERATIONAL SAFEGUARDS` block. Asked *"what happens if pass 1 of a merge run fails"*, the
 correct chunk scored **0.558** — below unrelated chunks from other files — because the other
 thirteen safeguards dominated the vector.
 The header convention writes each safeguard as a named paragraph: an unindented title, an
 indented body. Splitting on those titles took the same query to **0.718** and first place.
 ```
 Merge-Run Delete Interlock            ← its own chunk
  --delete is applied only when pass 1 completed...
 ```
 The parent section name is carried onto every sub-chunk, so routing still works. The title
 detection requires the *next* line to be indented — without that check, any wrapped prose line
 became a spurious boundary mid-sentence.
 ---
 ## ━━━ WHAT IS INDEXED ━━━
 Roughly 2,900 chunks across ~180 files:
 | Kind | Source |
 |------|--------|
 | `header` | Script headers — bash and PHP, sections and named paragraphs |
 | `readme` | Every `README-*.md` |
 | `manual` | Every `Manual-*.md` |
 | `template` | `Deployment/*.template` — the versioned conf schema |
 | `doc` | Top-level `README.md`, `Manual.md`, design notes |
 **Script bodies are not indexed.** Headers state intent, code states mechanism; for the
 questions this answers, intent retrieves better and costs far less.
 ---
 ## ━━━ THE SAFETY BOUNDARY ━━━
 **Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
 `Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
 was never in the repo to begin with. The index therefore describes the full conf schema (via the
 tracked templates, which carry all the explanatory comments) while structurally **never
 containing a secret**.
 > Do not "improve" this into a filesystem walk. A logged secret can be rotated. A secret
 > averaged into a 768-dimension float cannot be found, let alone removed.
 A live conf value the model genuinely needs should arrive through a tool call at query time,
 subject to redaction — never baked into a vector at index time.
 ---
 ## ━━━ HOW IT IS BUILT ━━━
 **SQLite, no vector database.** ~2,900 chunks × 768 dims is a couple of million multiply-adds
 per query — under a millisecond. A vector DB would be a container to run, monitor, fail over and
 back up, in exchange for nothing at this scale.
 **Vectors are raw little-endian float32 BLOBs.** `nomic-embed-text` returns L2-normalised
 vectors, so cosine similarity is a plain dot product — no normalising, no magnitude cache. PHP
 reads the same blobs with `unpack('f*', $blob)` when the UI needs them.
 **Incremental on mtime.** A file whose mtime has not moved is skipped without being read. A
 no-op refresh takes about 70 ms; a full rebuild takes a few minutes.
 **Node for the maths, bash for everything else.** The bash entry points own configuration,
 gating, locking and logging exactly as every other Varaverk job does. Node owns only float
 vector maths and SQLite BLOBs. Same split as `api_cache_writer.sh` and its PHP.
 > `lib/index.js` uses `node:sqlite`, which Node still marks experimental. It is used because it
 > needs no native compilation on Unraid. If a Node upgrade ever breaks it, the index is
 > regenerable in minutes — this is a disposable artefact, not a datastore.
 ---
 ## ━━━ USAGE ━━━
 ```bash
 # Build (needs AI_ENABLED=true)
 bash AI/ai_index.sh                  # incremental
 bash AI/ai_index.sh --force          # full rebuild
 bash AI/ai_index.sh --dry-run        # what would be indexed; contacts nothing
 bash AI/ai_index.sh --status         # size, counts, last build
 # Ask
 bash AI/ai_query.sh "what stops rsync and the mover running at once"
 bash AI/ai_query.sh --search "why are the cache writers lockless"
 bash AI/ai_query.sh --section=CONFIGURATION "which variable sets the mover grace period"
 bash AI/ai_query.sh --json "..."     # for other scripts
 ```
 **`--search` is the trustworthy mode.** It returns verbatim repo text with nothing generated.
 When the answer matters, use it — or read the sources the generated answer cites.
 ---
 ## ━━━ TRUST THE SOURCES, NOT THE PROSE ━━━
 The generation prompt instructs the model to answer only from retrieved context and to say what
 is missing rather than fill the gap from general knowledge. That instruction matters here more
 than usual: this repo's conventions are frequently *not* the conventional ones, and a confident
 generic answer about rsync, Docker or systemd is worse than no answer.
 It works — and its faithfulness cut both ways on the first real test. Asked which variable
 controls the mover's grace window, the model answered `MOVER_STOP_TIMEOUT`, "defaults to 30
 seconds", citing `mover_stop.sh › CONFIGURATION`. The variable was right. The 30 was wrong — the
 real value is 300 — and the model was quoting the header verbatim. **The header was stale.**
 That sweep then found six stale `(default: N)` claims across the repo, all since corrected. The
 lesson is the operating principle for this whole folder:
 > Retrieval is exactly as accurate as the documentation it points at. When an answer looks
 > wrong, check the cited source before blaming the model — it is usually reporting a real
 > problem in the repo.
 ---
 ## ━━━ WHAT THIS DOES NOT DO ━━━
 - **It does not write conf.** `AI_CONF_WRITE_ENABLED` exists in `master.conf` and is off, with
  an empty key whitelist. Nothing in this folder writes a setting.
 - **It does not make decisions.** No watchdog, cleanup or fallback path consults it. The
  per-feature `AI_ASSIST_*` toggles are all false and each one is earned separately.
 - **It does not index code.** Headers and docs only.
 - **It does not sync.** The index is host-local and gitignored. Each node builds its own.
 - **It is not required.** Every script runs identically with `AI_ENABLED=false`.
 See `Notes_AI-Design.md` at the repo root for the wider design — host resolution across the
 Tailscale mesh, per-feature rollout tiers, and the conf-write guardrails that would have to be
 built before any of that is enabled.
 ---
 ## ━━━ SCHEDULING ━━━
 Not scheduled by default. When you want the index to track the repo automatically, the natural
 home is after a successful pull — the only moment the corpus actually changes:
 ```bash
 # master.conf → DAILY_MAINTENANCE_SCRIPTS, after git_pull_execute.sh
 "AI/ai_index.sh"
 ```
 `AI_INDEX_ON_PULL` exists in `master.conf` as the intended gate for wiring this into
 `git_pull_execute.sh` directly. It is not yet consumed by anything — the variable is reserved,
 not live.
 An incremental run on an unchanged repo is ~70 ms, so a daily entry costs effectively nothing
 and a pull that changed twelve files costs a few seconds.
@@ -0,0 +1,244 @@
 #!/bin/bash
 # ==============================================================================================
 # ━━━ AI Retrieval Index Builder ━━━
 # ==============================================================================================
 #
 # PURPOSE
 # ==============================================================================================
 # Builds and refreshes the retrieval index over Varaverk's own documentation — every script
 # header, folder README and Manual, and the conf templates. Chunks each file on the section
 # boundaries the header convention already defines, embeds each chunk through Ollama, and
 # stores the vectors in SQLite for AI/ai_query.sh to search.
 #
 # The index is derived data. It is gitignored, host-local, and rebuildable from the repo in
 # minutes — nothing depends on it surviving.
 #
 # ==============================================================================================
 # OPERATIONAL MODEL
 # ==============================================================================================
 # Chunking mirrors the header convention rather than using a fixed token window:
 #
 #   bash headers      split on the six section names, then sub-split named-paragraph
 #                     safeguards and principles so one question finds one answer
 #   PHP headers       same six names plus the per-layer tails (EXPORTS, RENDERS, ...)
 #   markdown          split on headings
 #   conf templates    split on the ━━━ section rules
 #
 # Every chunk keeps its section name as a field, which is what lets a query about a safeguard
 # be steered toward OPERATIONAL SAFEGUARDS chunks before similarity is considered.
 #
 # Incremental by file mtime. A file whose mtime has not moved is skipped without being read,
 # so a routine refresh costs well under a second and a full rebuild costs a few minutes.
 #
 # Heavy lifting runs in Node — float vector maths and SQLite BLOBs are genuinely awkward in
 # bash. This follows the api_cache_writer.sh precedent: a bash shim owning config, gating,
 # locking and logging, in front of the language that fits the work.
 #
 # ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
 # Tracked Files Only
 #   Indexes exactly what `git ls-files` reports. Configurations/, State_Files/ and data/ are
 #   gitignored, so it is structurally impossible for a credential to reach the index — the
 #   files holding them were never in the repo. This must never become a filesystem walk: a
 #   secret written into a vector cannot be rotated back out of it.
 #
 # The Index Is Disposable
 #   Stored under DATA_DIR, gitignored, and never synced to a partner. Losing it costs one
 #   rebuild. Nothing reads it as a source of truth — it points at files, and the files are
 #   the truth.
 #
 # Documentation Is The Corpus, Not The Code
 #   Script bodies are not indexed. The headers state intent and the code states mechanism;
 #   for the questions this answers, intent retrieves far better and is far cheaper.
 #
 # Off By Default
 #   Does nothing unless AI_ENABLED is true. A node with AI off never pays for this.
 #
 # ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
 # Fail-Closed Gate
 #   Exits cleanly unless AI_ENABLED is exactly "true". Any other value, including unset,
 #   means off.
 #
 # Root Enforcement
 #   Writes into DATA_DIR alongside other Varaverk state.
 #
 # Concurrency Lock
 #   acquire_lock() prevents two indexers racing on the same database.
 #
 # Dependency Verification
 #   Verifies node and the Ollama endpoint before touching the database. A missing dependency
 #   is reported and exits non-zero rather than leaving a half-built index.
 #
 # Reachability Pre-flight
 #   Probes the resolved Ollama URL with AI_CONNECT_TIMEOUT before starting. An unreachable
 #   endpoint aborts immediately instead of failing once per batch across the whole corpus.
 #
 # Partial Failure Is Not Recorded As Success
 #   A file whose chunks all failed to embed keeps its previous rows and its old mtime, so the
 #   next run retries it. A run with any failed batch exits 3.
 #
 # Atomic Per-Run Write
 #   All database changes commit in one transaction. An interrupted run leaves the previous
 #   index intact rather than a partially rewritten one.
 #
 # Deleted Files Are Removed From The Index
 #   A file that has left the repo has its chunks deleted, so retrieval cannot cite something
 #   that no longer exists.
 #
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #
 # master.conf
 #
 #   AI_ENABLED
 #       Master switch. Fail-closed — must be exactly "true".
 #
 #   AI_INDEX_DB
 #       SQLite index path. (shipped default: $DATA_DIR/ai_index.db)
 #
 #   AI_INDEX_BATCH
 #       Chunks per embed request. (shipped default: 32)
 #
 #   AI_CONNECT_TIMEOUT
 #       Seconds for the reachability probe. (shipped default: 5)
 #
 #   AI_REQUEST_TIMEOUT
 #       Seconds for a single embed batch. (shipped default: 240)
 #
 # host*.conf
 #
 #   HOST*_OLLAMA_URL
 #       This host's Ollama endpoint. Empty means no local Ollama.
 #
 #   HOST*_OLLAMA_EMBED_MODEL
 #       Embedding model. The generation model cannot embed.
 #
 # ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
 #
 # ai_index.sh
 #     Incremental refresh — only files whose mtime moved are re-embedded.
 #
 # ai_index.sh --force
 #     Full rebuild. Discards the existing index and re-embeds every chunk.
 #
 # ai_index.sh --dry-run
 #     Report what would be indexed. Contacts nothing and writes nothing.
 #
 # ai_index.sh --status
 #     Show index location, size, chunk counts by kind and section, and last build time.
 #
 # ai_index.sh --log
 #     Verbose — per-batch embedding progress.
 #
 # ==============================================================================================
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "${SCRIPT_DIR}/../load_config.sh"
 detect_hosts
 DRY_RUN=false; FORCE=false; STATUS=false; LOG=false
 for arg in "$@"; do
    case "$arg" in
        --dry-run) DRY_RUN=true ;;
        --force)   FORCE=true ;;
        --status)  STATUS=true ;;
        --log)     LOG=true ;;
        *) echo "Unknown option: $arg" >&2; exit 1 ;;
    esac
 done
 CLI="${SCRIPT_DIR}/lib/cli.js"
 DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
 _url_var="${MY_ID}_OLLAMA_URL"
 _emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
 OLLAMA_URL="${!_url_var:-}"
 EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
 # ── Status ────────────────────────────────────────────────────────────────────────────────────
 if [[ "$STATUS" == true ]]; then
    echo "$ICON_GEAR  AI Index Status"
    echo "   Enabled:   ${AI_ENABLED:-false}"
    echo "   Database:  $DB"
    if [[ -f "$DB" ]]; then
        echo "   Size:      $(du -h "$DB" 2>/dev/null | cut -f1)"
        echo "   Chunks:    $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null || echo '?')"
        echo "   Files:     $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_files;'  2>/dev/null || echo '?')"
        _last=$(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='last_index';" 2>/dev/null)
        [[ -n "$_last" ]] && echo "   Last built: $(date -d "@$_last" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)"
        echo "   Model:     $(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='embed_model';" 2>/dev/null || echo '?')"
        echo "   By kind:"
        sqlite3 "$DB" "SELECT '     '||kind||': '||COUNT(*) FROM vv_chunks GROUP BY kind ORDER BY COUNT(*) DESC;" 2>/dev/null
    else
        echo "   Database:  not built yet"
    fi
    echo "   Ollama:    ${OLLAMA_URL:-<none local>}"
    echo "   Embed:     $EMBED_MODEL"
    exit 0
 fi
 # ── Gate ──────────────────────────────────────────────────────────────────────────────────────
 if [[ "${AI_ENABLED:-false}" != "true" ]]; then
    log "AI_ENABLED is not true — skipping index build"
    exit 0
 fi
 if [[ "$DRY_RUN" == false && "$EUID" -ne 0 ]]; then
    error "Must be run as root"
    exit 1
 fi
 acquire_lock
 command -v node >/dev/null 2>&1 || { error "node not found — required to build the index"; exit 1; }
 [[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
 if [[ -z "$OLLAMA_URL" ]]; then
    error "${MY_ID}_OLLAMA_URL is empty — no local Ollama to index against"
    exit 1
 fi
 # Pre-flight: fail once, up front, rather than once per batch across the whole corpus.
 if [[ "$DRY_RUN" == false ]]; then
    if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
        error "Ollama unreachable at $OLLAMA_URL"
        exit 1
    fi
 fi
 # ── Build ─────────────────────────────────────────────────────────────────────────────────────
 _args=(
    index
    "--root=${SCRIPTS_DIR}"
    "--db=${DB}"
    "--url=${OLLAMA_URL}"
    "--model=${EMBED_MODEL}"
    "--batch=${AI_INDEX_BATCH:-32}"
    "--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
 )
 [[ "$FORCE"   == true ]] && _args+=(--force)
 [[ "$DRY_RUN" == true ]] && _args+=(--dry-run)
 [[ "$LOG"     == false ]] && _args+=(--quiet)
 log "$ICON_GEAR Building AI index → $DB"
 node --no-warnings "$CLI" "${_args[@]}"
 _rc=$?
 case "$_rc" in
    0) log "$ICON_DONE AI index build complete" ;;
    3) warn "AI index built with some batches failed — those files will retry next run" ;;
    *) error "AI index build failed (exit $_rc)" ;;
 esac
 exit "$_rc"
@@ -0,0 +1,184 @@
 #!/bin/bash
 # ==============================================================================================
 # ━━━ AI Retrieval Query ━━━
 # ==============================================================================================
 #
 # PURPOSE
 # ==============================================================================================
 # Answers questions about Varaverk from Varaverk's own documentation. Embeds the question,
 # retrieves the closest chunks from the index AI/ai_index.sh built, and either prints them
 # directly or passes them to the generation model as grounding context.
 #
 # ==============================================================================================
 # OPERATIONAL MODEL
 # ==============================================================================================
 # Two modes over the same retrieval:
 #
 #   --search    print the matching chunks and their sources. No generation model involved,
 #               so it is fast and its output is verbatim repo text.
 #   (default)   retrieve, then ask the generation model to answer strictly from what was
 #               retrieved, citing each claim.
 #
 # Retrieval is steered by question shape. A question about what prevents something is pushed
 # toward OPERATIONAL SAFEGUARDS chunks, one about a variable toward CONFIGURATION, one asking
 # why toward DESIGN PRINCIPLES. This is a score boost, not a filter — a heuristic must not be
 # able to exclude the chunk that actually holds the answer.
 #
 # ==============================================================================================
 # DESIGN PRINCIPLES
 # ==============================================================================================
 #
 # Grounded Or Silent
 #   The prompt instructs the model to answer only from retrieved context and to say what is
 #   missing rather than fill the gap. This repo's conventions are frequently not the
 #   conventional ones, and a confident generic answer about rsync or Docker is worse here
 #   than no answer.
 #
 # Sources Are Always Shown
 #   Every answer prints the chunks it drew on. An answer that cannot be traced back to a file
 #   is not usable for changing anything.
 #
 # Search Is The Trustworthy Mode
 #   --search returns repo text with nothing generated. When an answer matters, use it.
 #
 # Read-Only
 #   Retrieves and answers. Nothing here writes conf, touches state, or runs another script.
 #
 # ==============================================================================================
 # OPERATIONAL SAFEGUARDS
 # ==============================================================================================
 #
 # Fail-Closed Gate
 #   Exits cleanly unless AI_ENABLED is exactly "true".
 #
 # No Root Required
 #   Reads the index and calls Ollama. Nothing it does needs privilege, so it does not ask for
 #   any — this is the one AI script an ordinary user should be able to run.
 #
 # Missing Index Is Reported, Not Built
 #   An absent index exits with guidance to run ai_index.sh. Building a corpus-wide index as a
 #   side effect of a question would turn a two-second query into a several-minute one.
 #
 # Reachability Pre-flight
 #   Probes Ollama with AI_CONNECT_TIMEOUT before embedding, so an unreachable endpoint fails
 #   immediately with a clear message.
 #
 # Bounded Generation
 #   The request is capped at AI_REQUEST_TIMEOUT. A wedged model cannot hang the caller.
 #
 # ==============================================================================================
 # CONFIGURATION
 # ==============================================================================================
 #
 # master.conf
 #
 #   AI_ENABLED
 #       Master switch. Fail-closed — must be exactly "true".
 #
 #   AI_INDEX_DB
 #       SQLite index to search. (shipped default: $DATA_DIR/ai_index.db)
 #
 #   AI_SEARCH_K
 #       Chunks retrieved per query. (shipped default: 8)
 #
 #   AI_SEARCH_PER_FILE
 #       Cap per file, so one document cannot fill the context. (shipped default: 3)
 #
 #   AI_REQUEST_TIMEOUT
 #       Seconds allowed for generation. (shipped default: 240)
 #
 # host*.conf
 #
 #   HOST*_OLLAMA_URL / HOST*_OLLAMA_MODEL / HOST*_OLLAMA_EMBED_MODEL
 #
 # ==============================================================================================
 # RUNTIME MODES
 # ==============================================================================================
 #
 # ai_query.sh "your question"
 #     Retrieve and answer, with sources.
 #
 # ai_query.sh --search "your question"
 #     Print matching chunks only. No generation.
 #
 # ai_query.sh --section=CONFIGURATION "your question"
 #     Restrict retrieval to one header section.
 #
 # ai_query.sh --json "your question"
 #     Machine-readable output for other scripts.
 #
 # ai_query.sh --status
 #     Show index and endpoint state.
 #
 # ==============================================================================================
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "${SCRIPT_DIR}/../load_config.sh"
 detect_hosts
 SEARCH_ONLY=false; JSON=false; STATUS=false; SECTION=""; QUERY=""
 for arg in "$@"; do
    case "$arg" in
        --search)     SEARCH_ONLY=true ;;
        --json)       JSON=true ;;
        --status)     STATUS=true ;;
        --section=*)  SECTION="${arg#*=}" ;;
        --*)          echo "Unknown option: $arg" >&2; exit 1 ;;
        *)            QUERY="$arg" ;;
    esac
 done
 CLI="${SCRIPT_DIR}/lib/cli.js"
 DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
 _url_var="${MY_ID}_OLLAMA_URL"
 _gen_var="${MY_ID}_OLLAMA_MODEL"
 _emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
 OLLAMA_URL="${!_url_var:-}"
 GEN_MODEL="${!_gen_var:-}"
 EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
 if [[ "$STATUS" == true ]]; then
    echo "$ICON_GEAR  AI Query Status"
    echo "   Enabled:  ${AI_ENABLED:-false}"
    echo "   Index:    $DB $([[ -f "$DB" ]] && echo "($(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null) chunks)" || echo '(not built)')"
    echo "   Ollama:   ${OLLAMA_URL:-<none>}"
    echo "   Generate: ${GEN_MODEL:-<unset>}"
    echo "   Embed:    $EMBED_MODEL"
    exit 0
 fi
 if [[ "${AI_ENABLED:-false}" != "true" ]]; then
    echo "AI_ENABLED is not true — AI features are off" >&2
    exit 0
 fi
 [[ -z "$QUERY" ]] && { echo "usage: ai_query.sh [--search] [--section=NAME] \"your question\"" >&2; exit 1; }
 [[ -f "$DB" ]]   || { error "No index at $DB — run AI/ai_index.sh first"; exit 1; }
 [[ -f "$CLI" ]]  || { error "missing $CLI"; exit 1; }
 command -v node >/dev/null 2>&1 || { error "node not found"; exit 1; }
 [[ -z "$OLLAMA_URL" ]] && { error "${MY_ID}_OLLAMA_URL is empty"; exit 1; }
 if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
    error "Ollama unreachable at $OLLAMA_URL"
    exit 1
 fi
 _args=("--db=${DB}" "--url=${OLLAMA_URL}" "--query=${QUERY}")
 [[ -n "$SECTION" ]] && _args+=("--section=${SECTION}")
 [[ "$JSON" == true ]] && _args+=(--json)
 if [[ "$SEARCH_ONLY" == true ]]; then
    node --no-warnings "$CLI" search "${_args[@]}" \
        "--model=${EMBED_MODEL}" \
        "--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}"
 else
    [[ -z "$GEN_MODEL" ]] && { error "${MY_ID}_OLLAMA_MODEL is empty — needed for generation"; exit 1; }
    node --no-warnings "$CLI" ask "${_args[@]}" \
        "--model=${GEN_MODEL}" "--embed-model=${EMBED_MODEL}" \
        "--k=${AI_SEARCH_K:-8}" "--per-file=${AI_SEARCH_PER_FILE:-3}" \
        "--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
 fi
@@ -122,13 +122,9 @@ async function buildIndex(opts) {
        let chunks = [];
        try { chunks = chunkFile(abs, rel); } catch (e) { continue; }
-        if (!chunks.length) {
+        // A file that yields no chunks still gets a vv_files row so it is not re-chunked every
-            if (db) {
+        // run. Queued rather than written here, so every database change lands in the single
-                db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(rel);
+        // commit below — a run interrupted mid-embed must leave the index exactly as it was.
                db.prepare('INSERT OR REPLACE INTO vv_files VALUES (?,?,?,?)').run(rel, mtime, 0, now);
            }
            continue;
        }
        pending.push({ rel, mtime, chunks });
    }
@@ -166,6 +162,12 @@ async function buildIndex(opts) {
    db.exec('BEGIN');
    try {
        for (const f of pending) {
            // Nothing to index in this file at all — record it so it is not re-chunked next run.
            if (!f.chunks.length) {
                db.prepare('DELETE FROM vv_chunks WHERE path = ?').run(f.rel);
                insF.run(f.rel, f.mtime, 0, now);
                continue;
            }
            const embedded = f.chunks.filter(c => c.__vec);
            // A file whose chunks all failed to embed keeps its previous rows and its old mtime,
            // so the next run retries it rather than recording a half-indexed file as current.
@@ -569,6 +569,20 @@
    HOSTN_LLDAP_USER="admin"                    # lldap admin username
    HOSTN_LLDAP_PASS=""                         # lldap admin password
 # ==============================================================================================
 # ── Ollama / AI ───────────────────────────────────────────────────────────────────────────────
 # ==============================================================================================
 # Per-host because only some nodes actually have a GPU. A node with an empty OLLAMA_URL is not
 # an error — it falls through to the resolver and uses another node's Ollama over Tailscale.
 # ━━━ Ollama ━━━
    HOSTN_OLLAMA_URL=""                          # e.g. http://localhost:11434 — empty if no local Ollama
    HOSTN_OLLAMA_CONTAINER="Ollama"              # for watchdog / restart lists
    HOSTN_OLLAMA_GPU_UUID=""                     # pins Ollama to one card on multi-GPU hosts
    HOSTN_OLLAMA_MODEL="qwen2.5-coder:14b"       # generation
    HOSTN_OLLAMA_EMBED_MODEL="nomic-embed-text"  # embeddings — the generation model cannot embed
 # ━━━ Authelia ━━━
    HOSTN_AUTHELIA_CONFIG="/mnt/user/appdata-Fallback/Critical-Data/Authelia/configuration.yml"
    HOSTN_AUTHELIA_CONTAINER="Authelia"
@@ -1593,6 +1593,54 @@
    SYS_WATCHDOG_ABORT_ON_PARITY=false           # aborting parity better than crashing mid-check
    SYS_WATCHDOG_ABORT_ON_MOVER=false            # aborting move better than crashing mid-move
 # ==============================================================================================
 # ── AI / RAG ──────────────────────────────────────────────────────────────────────────────────
 # ==============================================================================================
 # Varaverk works exactly as well with AI off as with it on. Nothing below is required for any
 # script to function — every feature that can lean on AI has a complete non-AI path, and the
 # AI path is an enhancement layered on top. If Ollama is unreachable, callers proceed without it.
 #
 # AI_ENABLED is necessary but not sufficient. Every feature stays individually off until it has
 # earned it — narration for months before anything is allowed near a decision.
 # ━━━ AI Master Switch ━━━
 # Fail-closed: anything other than the literal "true" means off.
    AI_ENABLED=false
    AI_CONNECT_TIMEOUT=5            # seconds — probe when resolving which node has Ollama
    AI_REQUEST_TIMEOUT=240          # seconds — must clear a cold model load
    AI_RESOLVE_CACHE_TTL=300        # seconds — don't re-probe the mesh every invocation
    AI_MAX_RETRIES=1                # AI is enhancement; do not retry hard
 # ━━━ AI Retrieval Index ━━━
 # The RAG index over this repo's own headers and documentation. Regenerable in minutes and
 # gitignored — it is derived data, never a source of truth.
 #
 # Only git-tracked files are ever indexed. Configurations/, State_Files/ and data/ are
 # gitignored, which is what makes it structurally impossible for a credential to reach the
 # index: the files holding them were never in the repo. Do not "improve" this to a filesystem
 # walk — an embedded secret cannot be rotated out of a vector.
    AI_INDEX_DB="$DATA_DIR/ai_index.db"
    AI_INDEX_BATCH=32               # chunks per embed request
    AI_INDEX_ON_PULL=false          # re-index after a successful git pull once AI is in use
    AI_SEARCH_K=8                   # chunks retrieved per query
    AI_SEARCH_PER_FILE=3            # cap per file so one document cannot fill the context
 # ━━━ AI Feature Toggles ━━━
 # Tier 1 is narration — it cannot change a decision. Tier 2 adds context to a decision a script
 # already made. Tier 3 assists a human. Enable in that order, and give each one weeks.
    AI_ASSIST_REPORTS=false         # tier 1 — digest / coffee report narration
    AI_ASSIST_WATCHDOG=false        # tier 2 — context on a flagged condition
    AI_ASSIST_DISCOVERY=false       # tier 2 — discovery / classification judgement calls
    AI_ASSIST_CLEANUP=false         # tier 2 — orphan and stuck-import triage
    AI_ASSIST_ONBOARD=false         # tier 3 — onboarding / settings assistance
 # ━━━ AI Conf Write Access ━━━
 # Separate switch from AI_ENABLED, off by default, and an explicit key whitelist. Never paths,
 # never credentials, never a container name. An empty whitelist means no writes regardless of
 # the toggle.
    AI_CONF_WRITE_ENABLED=false
    AI_CONF_WRITE_KEYS=()
 # ==============================================================================================
 # ──────────────────────── End Of User Variables ───────────────────────────────────────────────
 # ==============================================================================================
@@ -1,7 +1,13 @@
 # Varaverk AI Integration — Design Notes
-**Status: design only. Nothing below is built.** No Varaverk script calls Ollama, and no
+**Status: retrieval is built; integration is not.** As of 2026-08-02 the `AI_*` and
-`AI_*` variable exists in any conf yet. Captured 2026-08-01 so the reasoning survives.
+`HOST*_OLLAMA_*` variables exist in both confs and both templates, and `AI/` holds a working
 index and query path — see the RAG section at the end of this document and `AI/README-AI.md`.
 Everything else below remains design only. **No Varaverk script consults AI.** Every
 `AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false with an empty whitelist, and
 host resolution across the mesh is specified but not implemented. Originally captured
 2026-08-01 so the reasoning survives.
 Ollama itself *is* installed, tuned and verified on HOST1 — `qwen2.5-coder:14b` for
 generation, `nomic-embed-text` for embeddings, 16k context, pinned to the RTX 3080. See
@@ -490,3 +496,64 @@ new — the machinery already exists and was audited this session.
  possibly credentials pasted by a user.
 - Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the
  serialisation? Defer until an actual problem is felt.
 ---
 ## RAG — built 2026-08-02
 Retrieval is live. `AI/` holds the implementation; `AI/README-AI.md` documents it in full. What
 follows is only what changed relative to the plan recorded above.
 **Corpus is larger than estimated.** ~2,950 chunks across ~180 files, not the 690 header chunks
 projected. Sub-chunking is why — see below.
 **Named-paragraph sub-chunking was necessary, and was not in the plan.** Section-level chunks
 alone were too coarse. `rsync.sh` documents fourteen safeguards in one 2.8k-char
 `OPERATIONAL SAFEGUARDS` block; a query about one of them scored 0.558, below unrelated chunks,
 because the other thirteen dominated the vector. Splitting on the named-paragraph titles the
 header convention already uses took the same query to 0.718 and first place. The parent section
 name is carried onto each sub-chunk so routing still works.
 **Two chunker bugs worth remembering.** The last section in a header (RUNTIME MODES in bash,
 DEPENDS ON in a page) ran to EOF and swept up every unrelated comment in the file —
 `scheduler.php` alone produced an 11k-char chunk of unrelated inline comments. And title
 detection must require the *next* line to be indented; without that, any wrapped prose line
 became a spurious chunk boundary mid-sentence.
 **Section routing is a boost, not a filter.** Intent detection is a heuristic and must not be
 able to exclude the chunk holding the answer. `--section=` forces a hard filter when wanted.
 **Vectors arrive pre-normalised.** `nomic-embed-text` returns L2-normalised vectors (measured
 norm 1.0000001), so cosine is a plain dot product. No normalising step, no magnitude cache.
 **`node:sqlite` over a native module.** Still flagged experimental, chosen because it needs no
 native compilation on Unraid. Acceptable because the index is disposable — if a Node upgrade
 breaks it, rebuild takes minutes. PHP reads the same float32 blobs with `unpack('f*', $blob)`
 when the UI needs them.
 **Retrieval quality, measured.** 9/10 top-3 hit rate on known-answer questions; the tenth had
 the answer at ranks 2 and 3, so 10/10 for answer-present-in-context at k=6. Full retrieval plus
 generation runs about 43s warm.
 ### The finding that validated the whole thing
 First real end-to-end question asked which variable controls the mover's grace window. The model
 answered `MOVER_STOP_TIMEOUT`, "defaults to 30 seconds", citing `mover_stop.sh › CONFIGURATION`.
 Variable correct; the 30 was wrong — the real value is 300. The model was quoting the header
 verbatim. **The header was stale.**
 A sweep for the same pattern found six stale `(default: N)` claims across the repo — all
 corrected in the same pass. This is the operating principle for the folder:
 > Retrieval is exactly as accurate as the documentation it points at. When an answer looks
 > wrong, check the cited source before blaming the model.
 It also means the index is a documentation-drift detector, not only a question-answering tool.
 ### Still deliberately not built
 Nothing consults this. Every `AI_ASSIST_*` toggle is false, `AI_CONF_WRITE_ENABLED` is false
 with an empty key whitelist, and no watchdog, cleanup or fallback path calls it. Host resolution
 across the Tailscale mesh is specified above but not implemented — `ai_index.sh` and
 `ai_query.sh` currently require a local `HOST*_OLLAMA_URL` and fail with a clear message when it
 is empty, rather than silently probing the mesh.