From 986353a0f00fc09b7409d5515a15ee535a633c03 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 1 Aug 2026 21:30:08 -0400 Subject: [PATCH] Add AI integration design notes Captures the reasoning behind a not-yet-built feature so the constraints survive the session, chiefly that AI stays enhancement-only and never load-bearing. --- Notes_AI-Design.md | 340 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 340 insertions(+) create mode 100644 Notes_AI-Design.md diff --git a/Notes_AI-Design.md b/Notes_AI-Design.md new file mode 100644 index 0000000..a497418 --- /dev/null +++ b/Notes_AI-Design.md @@ -0,0 +1,340 @@ +# Varaverk AI Integration — Design Notes + +**Status: design only. Nothing below is built.** Ollama is installed and running on HOST1; +no Varaverk script calls it. Captured 2026-08-01 so the reasoning survives. + +**Origin:** the RTX 3080 was freed when the Windows gaming VM was retired. It is bound to the +`nvidia` driver, not `vfio` — not reserved for passthrough, so there is no VM contention to +design around. Two goals at once: somewhere to learn local LLMs, and something Varaverk can +genuinely use. + +**Build order** — deliberately lowest-risk first. Each stage must be boring before the next +one starts: + +1. Chat assistant / settings helper / onboarding assistant — a wrong answer costs nothing +2. Watchdog and discovery context — a wrong answer costs a bad suggestion, still gated +3. Cleanup and sync decision aid — closest to destructive, last to be trusted + +The regular script is always the backup, at every stage. + +--- + +## The governing principle + +**Varaverk works exactly as well with AI off as with it on.** + +Every script — including scripts written after this lands — is designed and hardened +without AI first. AI is added afterwards as enhancement, never as a dependency. A script +that cannot do its job when `AI_ENABLED=false` is a broken script, not an AI feature. + +This is the constraint everything else in this document answers to. If a design decision +makes AI load-bearing, that decision is wrong. + +Corollary: AI never makes a destructive decision. The session that produced the current +safeguard layer (depth guards, strike thresholds, verification-after-write) exists because +config values and scan results feed `rm -rf`, `chown -R`, and rsync `--delete`. AI advises +at the points where a script currently stops and defers to a human. The deterministic guard +still pulls the trigger. + +--- + +## Two independent off-switches + +| switch | meaning | source | +|---|---|---| +| `AI_ENABLED` | intent — do we want AI at all | `master.conf` | +| resolver result | availability — is there a reachable host | runtime probe | + +**Both must produce the identical code path when off.** A caller that gets "no AI" from +either must run its normal, non-AI logic — not a degraded variant, not a skipped step. + +`AI_ENABLED` follows the fail-closed idiom standardised across the ecosystem: + +```bash +[[ "${AI_ENABLED:-false}" != "true" ]] && +``` + +Not `== false`. Anything that isn't exactly `true` means off, so a typo can never switch AI +on. (This is the same bug that was fixed in `fallback.sh` — see its `FALLBACK_ENABLED` gate.) + +--- + +## Configuration schema + +Follows the existing rule: **thresholds and toggles → `master.conf`; hardware, paths, +container names and per-host identity → `host*.conf`.** + +### `host*.conf` — per-host, because only one node actually has the GPU + +```bash +# ━━━ Ollama / AI ━━━ + HOST1_OLLAMA_URL="http://localhost:11434" # empty on nodes without a local Ollama + HOST1_OLLAMA_CONTAINER="Ollama" # for docker_watchdog / restart lists + HOST1_OLLAMA_GPU_UUID="GPU-309357d8-2a13-09e0-84ac-fcfdcdf5c626" + HOST1_OLLAMA_MODEL="qwen2.5-coder:14b" # generation + HOST1_OLLAMA_EMBED_MODEL="nomic-embed-text" # embeddings — qwen cannot embed +``` + +A node with an empty `OLLAMA_URL` is not an error — it falls through to the resolver and +uses the mesh. HOST2 gets the section with blanks, exactly like the NPM/lldap credentials +it is already waiting on. + +### `master.conf` — shared behaviour + +```bash +# ━━━ AI ━━━ + AI_ENABLED=false # master switch — fail-closed, != "true" means off + AI_CONNECT_TIMEOUT=5 # probe timeout when resolving a host + AI_REQUEST_TIMEOUT=120 # generation can be slow; cold load was 28.6s + AI_RESOLVE_CACHE_TTL=300 # don't re-probe the mesh on every script invocation + AI_MAX_RETRIES=1 # AI is enhancement — do not retry hard + + # Per-feature toggles — enable narration long before enabling decision aid + AI_ASSIST_REPORTS=false # tier 1 — digest / coffee report narration + AI_ASSIST_WATCHDOG=false # tier 2 — context on a flagged condition + AI_ASSIST_DISCOVERY=false # tier 2 — discovery / classification judgment calls + AI_ASSIST_CLEANUP=false # tier 2 — HELD orphans, stuck-import triage + AI_ASSIST_ONBOARD=false # tier 3 — onboarding / settings assistance + + # Conf writes — separate switch, off by default, see Conf Write Access below + AI_CONF_WRITE_ENABLED=false + AI_CONF_WRITE_KEYS=() # explicit whitelist; never paths or credentials +``` + +**Per-feature toggles are load-bearing, not decoration.** They are what lets AI narrate the +weekly digest for months before it is ever allowed near a cleanup decision. `AI_ENABLED` is +necessary but not sufficient — every feature stays individually off until it has earned it. + +> When these land, `Deployment/master.conf.template` and `Deployment/host.conf.template` +> must be updated in the same pass. That rule is not optional in this repo. + +--- + +## Host resolution + +AI runs on the owner's node only. Remote mesh nodes reach it over Tailscale. No remote node +needs a GPU, a model, or an Ollama container — only the resolver. + +`resolve_ollama_host()` mirrors the existing Gitea locator in `git_pull_execute.sh`: + +``` +Ollama answering on localhost:11434? → use it (owner's node) + else discover_remote_nodes() + → resolve_tailscale_ip(node) + → probe each :11434 → first responsive wins + else → no AI host (== AI_ENABLED=false) +``` + +Helpers already exist in `common.sh`: `discover_remote_nodes()` (767), +`resolve_tailscale_ip()` (812), `check_connectivity()` (866). + +**Probe the API, not the container.** The Gitea locator checks `docker ps`. Do not copy that +here — a container can be up while the model is unloaded, still pulling, or wedged. Probe +`/api/tags`. Same principle written into `network_watchdog.sh`'s design principles: +*verify the path, not the process*. + +**Cache the resolution** in `/tmp` state, like the arr cache. A cleanup script should not pay +a Tailscale round-trip to discover AI it may never call. + +### Known consequence + +AI lives on HOST1, so **during a fallback — HOST1 down, HOST2 covering — the mesh has no AI.** +That is precisely when a triage assistant would be most useful. Accepted: a second GPU on +HOST2 is a lot of hardware for that window, and AI is enhancement-only by design. Worth +knowing rather than discovering. + +--- + +## Where AI is allowed to act + +Ranked by how much damage a wrong answer does. + +**Tier 1 — narration and summary (safe, do first)** +- Sunday morning coffee report — turn metrics into prose +- `weekly_health_digest.sh` — summarise, highlight what changed +- Explain *why* a container is crash-looping from its logs + +**Tier 2 — triage and context on an existing flag (the real value)** +Places where a script already detects something and stops: +- `system_watchdog` / `stability_watchdog` flags a condition → AI adds context, correlates + with recent logs, suggests likely cause +- Sonarr stuck-import triage — the "matched by series ID" recipe is textbook LLM work +- `HELD` entries from `arr_download_orphan_cleaner.sh` +- `reverse-anime-leak` from the classification scans — currently report-only *because* it is + a judgment call. That is exactly the shape AI suits. + +**Tier 3 — assisted configuration (needs the guardrails below)** +- Onboarding a new host — the main motivation for conf write access +- AI-assisted settings tuning: rsync profiles, fallback tiers, auth stack + +**Never** +- Deciding what to delete +- Choosing a path for any destructive operation +- Anything that bypasses a strike counter, age gate, or verification step + +--- + +## Conf write access + +Wanted mainly for onboarding and assisted settings. This is the highest-risk item here. + +**Current state: there is no recovery path.** + +``` +.gitignore:4 Configurations/host*.conf +.gitignore:5 Configurations/master.conf +.gitignore:6 Configurations/*.bak +``` + +Confs are gitignored — no git history to revert to — and so are the `.bak` files, so the +backup is not versioned either. The only fallback is a single `.bak` slot written by +`conf_upgrade`, and it goes stale immediately: + +| file | modified | its `.bak` | +|---|---|---| +| `master.conf` | Jul 28 18:52 | Jul 28 18:52 | +| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** | + +A bad write to `master.conf` currently falls back to a file that may predate a month of edits. +**Fix this before any AI writes anything.** + +### Required before conf-write ships + +1. **Key whitelist, not file access.** Thresholds and toggles only — `*_WARN_GB`, + `*_STRIKE_LIMIT`, `*_ENABLED`, retention days. Never a path, never a credential, never a + container list. A wrong threshold is recoverable; a wrong path is what the depth guards + exist to catch. +2. **Timestamped backups, plural** — `master.conf.2026-08-01T21:00`, retained. Not one + clobbered slot. +3. **Validate before commit** — `bash -n` the candidate, then confirm `load_config.sh` + sources it cleanly. Never install a conf that has not been proven to parse. +4. **Diff always logged.** An AI conf change should be at least as visible as a container + restart. +5. **Lock against concurrent readers** — never rewrite a conf while scripts are mid-run. + +**Consider un-ignoring `Configurations/` into a private repo.** Then `git diff` and +`git revert` become the recovery mechanism and the history is free. This overlaps the +existing GitHub-mirror TODO, which is already blocked on the same question. + +--- + +## Security + +Ollama has **no authentication of any kind**, and its API includes `DELETE /api/delete` +(wipe models) and `POST /api/pull` (fill the disk). It currently binds `0.0.0.0:11434` with +`OLLAMA_ORIGINS=*` — reachable from the entire LAN, not just Tailscale. + +The design only needs loopback (owner) plus the Tailscale interface (mesh). `0.0.0.0` is +strictly wider than required, for no benefit. Restrict to loopback + Tailscale, or use +Tailscale ACLs to allow only mesh nodes. Node-level ACLs fit the mesh model better than +app-level auth Ollama cannot provide anyway. + +Also set: `/ext-varaverk` mount is now `ro` in the template (was `rw` into live prod). +**Requires an Apply in the Docker tab to take effect on the running container.** + +--- + +## Hardware budget + +RTX 3080, 10 GB, pinned to Ollama by UUID — isolated from the Quadro P2000 that Emby +transcodes on. Do not let AI onto the P2000. + +| | VRAM | +|---|---| +| `qwen2.5-coder:14b` (Q4_K_M) | 8.34 GB | +| `nomic-embed-text` (768-dim) | 0.25 GB | +| **used / total** | **9.30 / 10.24 GB (91%)** | + +Latency: 28.6s cold load, ~2.6s warm, ~4.4s warm with both models resident. +`OLLAMA_KEEP_ALIVE=-1` keeps both pinned, so cold load is a boot-time cost only. + +**You cannot have 14B + long context + real parallelism on 10 GB.** Pick two. +`OLLAMA_MAX_QUEUE=512` means excess requests queue rather than fail, and at ~4s responses, +two or three users serialised is barely noticeable. Multi-user hits are expected to be rare. +So: keep parallelism low, let the queue absorb bursts, spend VRAM on **context** — that is +what RAG actually needs. + +Tuning order matters. At 91% utilisation, raising context first will OOM: + +1. `OLLAMA_NUM_PARALLEL` 2 → 1 (frees KV cache) +2. `OLLAMA_KV_CACHE_TYPE` f16 → q8_0 (roughly halves KV memory, negligible quality cost) +3. *then* `OLLAMA_CONTEXT_LENGTH` 4096 → 8192/16384 + +All three are env vars in the same template, so they ride along with the same Apply as the +`ro` mount fix — one container recreate covers everything. + +--- + +## RAG + +No Python on Unraid, and none needed. Everything required is already present: +`sqlite3 3.53`, `jq 1.8`, `node 22`, `php 8.4`, `awk`. + +- **chunks + vectors** → SQLite, one table +- **similarity** → cosine in PHP or Node; milliseconds at this corpus size, no vector DB + container needed +- **embed + generate** → Ollama HTTP, same `curl` pattern as every other integration here + +Corpus: **49,654 lines of bash + ~11,970 lines of markdown.** Small. + +**Chunking is already solved.** Every script now carries +`PURPOSE / OPERATIONAL MODEL / DESIGN PRINCIPLES / OPERATIONAL SAFEGUARDS / CONFIGURATION / +RUNTIME MODES` at exact, greppable boundaries. Those are semantically coherent units with +stable headings — far better retrieval chunks than fixed-size windows. The `DESIGN PRINCIPLES` +sections encode *why*, which is what the model needs and what code alone never says. + +Note `qwen2.5-coder` returns `501 — does not support embeddings`. Embedding is +`nomic-embed-text`'s job. Batch embedding works (n inputs → n vectors in one call) and is +required — indexing 50k lines one HTTP call at a time is not viable. + +--- + +## Scheduled AI + +Same orchestrator tiers as everything else, gated on `AI_ENABLED` plus a reachable host. +Natural fits: weekly digest narration, a periodic pass over `HELD`/report-only findings that +have accumulated, post-incident summaries after a watchdog event. + +Must obey the existing tier discipline — an AI job that fails or times out is a non-fatal +step like any other, and never blocks the rest of its tier. + +--- + +## UI + +- **Dedicated AI page** in the plugin. +- **Persistent conversation across pages.** A floating widget is not required — a chat column + is fine. The constraint is that Unraid's WebGUI is multi-page PHP with full reloads and no + SPA shell, so persistence means conversation state lives server-side keyed by session, with + the client re-hydrating per page. +- **Per-page AI helpers** — contextual assistance scoped to whatever that page is about. + +--- + +## "Too bad we can't just run the LLM inside Varaverk and cut out Ollama" + +There is a real answer: you don't cut Ollama out, you **absorb it**. + +Ollama does non-trivial work — model lifecycle, GPU scheduling, keep-alive, batching, an HTTP +API. Reimplementing that in bash is not a good trade. But Varaverk already manages containers +better than most things manage containers. Ollama becomes just another managed container: + +- add to `HOST*_WATCHDOG_CONTAINERS` so `docker_watchdog.sh` keeps it healthy +- add to a restart list so it gets the same proactive treatment as everything else +- give it a fallback tier if AI should survive a host outage +- let `docker_update.sh` handle its image updates + +That is more Varaverk-native than embedding a model runtime would be, and it costs nothing +new — the machinery already exists and was audited this session. + +--- + +## Open questions + +- Un-ignore `Configurations/` into a private repo for conf history? (blocks conf-write, and + overlaps the GitHub-mirror TODO) +- Does the AI page need auth separate from the Unraid WebGUI, given remote mesh members? +- Retention/privacy for conversation history — logs may contain paths, container names, + possibly credentials pasted by a user. +- Is a 7B worth it to buy context + parallelism headroom, or is 14B quality worth the + serialisation? Defer until an actual problem is felt.