Add AI entry points, conf schema, and folder docs
ai_index.sh and ai_query.sh follow the usual conventions — fail-closed gate, root check, lock, dry-run, status — with Node doing only the vector maths and SQLite blobs, the same split api_cache_writer.sh uses for PHP. AI_* and HOST*_OLLAMA_* land in both confs and both templates in this pass. Everything ships off: AI_ENABLED false, every AI_ASSIST_* false, conf writes disabled with an empty whitelist. Nothing in the ecosystem consults it.
This commit is contained in:
Gmer4Lfe
Executable
+244
@@ -0,0 +1,244 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ━━━ AI Retrieval Index Builder ━━━
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ==============================================================================================
|
||||
# Builds and refreshes the retrieval index over Varaverk's own documentation — every script
|
||||
# header, folder README and Manual, and the conf templates. Chunks each file on the section
|
||||
# boundaries the header convention already defines, embeds each chunk through Ollama, and
|
||||
# stores the vectors in SQLite for AI/ai_query.sh to search.
|
||||
#
|
||||
# The index is derived data. It is gitignored, host-local, and rebuildable from the repo in
|
||||
# minutes — nothing depends on it surviving.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
# Chunking mirrors the header convention rather than using a fixed token window:
|
||||
#
|
||||
# bash headers split on the six section names, then sub-split named-paragraph
|
||||
# safeguards and principles so one question finds one answer
|
||||
# PHP headers same six names plus the per-layer tails (EXPORTS, RENDERS, ...)
|
||||
# markdown split on headings
|
||||
# conf templates split on the ━━━ section rules
|
||||
#
|
||||
# Every chunk keeps its section name as a field, which is what lets a query about a safeguard
|
||||
# be steered toward OPERATIONAL SAFEGUARDS chunks before similarity is considered.
|
||||
#
|
||||
# Incremental by file mtime. A file whose mtime has not moved is skipped without being read,
|
||||
# so a routine refresh costs well under a second and a full rebuild costs a few minutes.
|
||||
#
|
||||
# Heavy lifting runs in Node — float vector maths and SQLite BLOBs are genuinely awkward in
|
||||
# bash. This follows the api_cache_writer.sh precedent: a bash shim owning config, gating,
|
||||
# locking and logging, in front of the language that fits the work.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Tracked Files Only
|
||||
# Indexes exactly what `git ls-files` reports. Configurations/, State_Files/ and data/ are
|
||||
# gitignored, so it is structurally impossible for a credential to reach the index — the
|
||||
# files holding them were never in the repo. This must never become a filesystem walk: a
|
||||
# secret written into a vector cannot be rotated back out of it.
|
||||
#
|
||||
# The Index Is Disposable
|
||||
# Stored under DATA_DIR, gitignored, and never synced to a partner. Losing it costs one
|
||||
# rebuild. Nothing reads it as a source of truth — it points at files, and the files are
|
||||
# the truth.
|
||||
#
|
||||
# Documentation Is The Corpus, Not The Code
|
||||
# Script bodies are not indexed. The headers state intent and the code states mechanism;
|
||||
# for the questions this answers, intent retrieves far better and is far cheaper.
|
||||
#
|
||||
# Off By Default
|
||||
# Does nothing unless AI_ENABLED is true. A node with AI off never pays for this.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Fail-Closed Gate
|
||||
# Exits cleanly unless AI_ENABLED is exactly "true". Any other value, including unset,
|
||||
# means off.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Writes into DATA_DIR alongside other Varaverk state.
|
||||
#
|
||||
# Concurrency Lock
|
||||
# acquire_lock() prevents two indexers racing on the same database.
|
||||
#
|
||||
# Dependency Verification
|
||||
# Verifies node and the Ollama endpoint before touching the database. A missing dependency
|
||||
# is reported and exits non-zero rather than leaving a half-built index.
|
||||
#
|
||||
# Reachability Pre-flight
|
||||
# Probes the resolved Ollama URL with AI_CONNECT_TIMEOUT before starting. An unreachable
|
||||
# endpoint aborts immediately instead of failing once per batch across the whole corpus.
|
||||
#
|
||||
# Partial Failure Is Not Recorded As Success
|
||||
# A file whose chunks all failed to embed keeps its previous rows and its old mtime, so the
|
||||
# next run retries it. A run with any failed batch exits 3.
|
||||
#
|
||||
# Atomic Per-Run Write
|
||||
# All database changes commit in one transaction. An interrupted run leaves the previous
|
||||
# index intact rather than a partially rewritten one.
|
||||
#
|
||||
# Deleted Files Are Removed From The Index
|
||||
# A file that has left the repo has its chunks deleted, so retrieval cannot cite something
|
||||
# that no longer exists.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# AI_ENABLED
|
||||
# Master switch. Fail-closed — must be exactly "true".
|
||||
#
|
||||
# AI_INDEX_DB
|
||||
# SQLite index path. (shipped default: $DATA_DIR/ai_index.db)
|
||||
#
|
||||
# AI_INDEX_BATCH
|
||||
# Chunks per embed request. (shipped default: 32)
|
||||
#
|
||||
# AI_CONNECT_TIMEOUT
|
||||
# Seconds for the reachability probe. (shipped default: 5)
|
||||
#
|
||||
# AI_REQUEST_TIMEOUT
|
||||
# Seconds for a single embed batch. (shipped default: 240)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_OLLAMA_URL
|
||||
# This host's Ollama endpoint. Empty means no local Ollama.
|
||||
#
|
||||
# HOST*_OLLAMA_EMBED_MODEL
|
||||
# Embedding model. The generation model cannot embed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ai_index.sh
|
||||
# Incremental refresh — only files whose mtime moved are re-embedded.
|
||||
#
|
||||
# ai_index.sh --force
|
||||
# Full rebuild. Discards the existing index and re-embeds every chunk.
|
||||
#
|
||||
# ai_index.sh --dry-run
|
||||
# Report what would be indexed. Contacts nothing and writes nothing.
|
||||
#
|
||||
# ai_index.sh --status
|
||||
# Show index location, size, chunk counts by kind and section, and last build time.
|
||||
#
|
||||
# ai_index.sh --log
|
||||
# Verbose — per-batch embedding progress.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "${SCRIPT_DIR}/../load_config.sh"
|
||||
|
||||
detect_hosts
|
||||
|
||||
DRY_RUN=false; FORCE=false; STATUS=false; LOG=false
|
||||
for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--dry-run) DRY_RUN=true ;;
|
||||
--force) FORCE=true ;;
|
||||
--status) STATUS=true ;;
|
||||
--log) LOG=true ;;
|
||||
*) echo "Unknown option: $arg" >&2; exit 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
CLI="${SCRIPT_DIR}/lib/cli.js"
|
||||
DB="${AI_INDEX_DB:-${DATA_DIR}/ai_index.db}"
|
||||
|
||||
_url_var="${MY_ID}_OLLAMA_URL"
|
||||
_emb_var="${MY_ID}_OLLAMA_EMBED_MODEL"
|
||||
OLLAMA_URL="${!_url_var:-}"
|
||||
EMBED_MODEL="${!_emb_var:-nomic-embed-text}"
|
||||
|
||||
# ── Status ────────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "$STATUS" == true ]]; then
|
||||
echo "$ICON_GEAR AI Index Status"
|
||||
echo " Enabled: ${AI_ENABLED:-false}"
|
||||
echo " Database: $DB"
|
||||
if [[ -f "$DB" ]]; then
|
||||
echo " Size: $(du -h "$DB" 2>/dev/null | cut -f1)"
|
||||
echo " Chunks: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_chunks;' 2>/dev/null || echo '?')"
|
||||
echo " Files: $(sqlite3 "$DB" 'SELECT COUNT(*) FROM vv_files;' 2>/dev/null || echo '?')"
|
||||
_last=$(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='last_index';" 2>/dev/null)
|
||||
[[ -n "$_last" ]] && echo " Last built: $(date -d "@$_last" '+%Y-%m-%d %H:%M:%S' 2>/dev/null)"
|
||||
echo " Model: $(sqlite3 "$DB" "SELECT v FROM vv_meta WHERE k='embed_model';" 2>/dev/null || echo '?')"
|
||||
echo " By kind:"
|
||||
sqlite3 "$DB" "SELECT ' '||kind||': '||COUNT(*) FROM vv_chunks GROUP BY kind ORDER BY COUNT(*) DESC;" 2>/dev/null
|
||||
else
|
||||
echo " Database: not built yet"
|
||||
fi
|
||||
echo " Ollama: ${OLLAMA_URL:-<none local>}"
|
||||
echo " Embed: $EMBED_MODEL"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Gate ──────────────────────────────────────────────────────────────────────────────────────
|
||||
if [[ "${AI_ENABLED:-false}" != "true" ]]; then
|
||||
log "AI_ENABLED is not true — skipping index build"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == false && "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
command -v node >/dev/null 2>&1 || { error "node not found — required to build the index"; exit 1; }
|
||||
[[ -f "$CLI" ]] || { error "missing $CLI"; exit 1; }
|
||||
|
||||
if [[ -z "$OLLAMA_URL" ]]; then
|
||||
error "${MY_ID}_OLLAMA_URL is empty — no local Ollama to index against"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pre-flight: fail once, up front, rather than once per batch across the whole corpus.
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
if ! curl -sf --max-time "${AI_CONNECT_TIMEOUT:-5}" "${OLLAMA_URL%/}/api/tags" >/dev/null 2>&1; then
|
||||
error "Ollama unreachable at $OLLAMA_URL"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Build ─────────────────────────────────────────────────────────────────────────────────────
|
||||
_args=(
|
||||
index
|
||||
"--root=${SCRIPTS_DIR}"
|
||||
"--db=${DB}"
|
||||
"--url=${OLLAMA_URL}"
|
||||
"--model=${EMBED_MODEL}"
|
||||
"--batch=${AI_INDEX_BATCH:-32}"
|
||||
"--timeout=$(( ${AI_REQUEST_TIMEOUT:-240} * 1000 ))"
|
||||
)
|
||||
[[ "$FORCE" == true ]] && _args+=(--force)
|
||||
[[ "$DRY_RUN" == true ]] && _args+=(--dry-run)
|
||||
[[ "$LOG" == false ]] && _args+=(--quiet)
|
||||
|
||||
log "$ICON_GEAR Building AI index → $DB"
|
||||
node --no-warnings "$CLI" "${_args[@]}"
|
||||
_rc=$?
|
||||
|
||||
case "$_rc" in
|
||||
0) log "$ICON_DONE AI index build complete" ;;
|
||||
3) warn "AI index built with some batches failed — those files will retry next run" ;;
|
||||
*) error "AI index build failed (exit $_rc)" ;;
|
||||
esac
|
||||
|
||||
exit "$_rc"
|
||||
Reference in New Issue
Block a user