Document today's restructures, and give data/ a README that ships with it

.gitignore

@@ -10,7 +10,11 @@ Configurations/*.bak
 Notes_To-Do.md
 
 # ── Runtime state, data, logs ─────────────────────────────────────────────────
-data/
+# Contents, not the directory itself. Ignoring "data/" outright means git never descends into
+# it, and a negation for a file inside an excluded directory is silently ineffective — so the
+# README explaining what data/ is would be the one file missing from every installation of it.
+data/*
+!data/README.md
 State_Files/
 .cache/
 *.log

AI/README-AI.md

@@ -119,7 +119,7 @@ questions this answers, intent retrieves better and costs far less.
 
 **Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
 
-`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
+`Configurations/` and `data/` are gitignored, so every file holding a credential
 was never in the repo to begin with. The index therefore describes the full conf schema (via the
 tracked templates, which carry all the explanatory comments) while structurally **never
 containing a secret**.
@@ -228,9 +228,54 @@ and a pull that changed twelve files costs a few seconds.
 
 ---
 
+## ━━━ PROFILES ━━━
+
+A profile is a contract plus a set of inputs. `Plugin/unraid/include/ai_profiles.php` is the one
+definition of both, read by the endpoint, the worker, the shared chat include and the Scheduler
+dock.
+
+| Profile | Turns | Retrieves | Notes |
+|---|---|---|---|
+| `varaverk` | 3 | yes | answers only from the index, with citations. The default. |
+| `chat` | 8 | **no** | ordinary conversation. Holds zero capabilities, deliberately. |
+| `code` | 4 | no | drafts shell for Custom Scripts; scans its own output for destructive ops |
+| `troubleshoot` | 2 | yes | reasons from an open log first, docs second. May file bug reports. |
+
+Capabilities are granted per profile — retrieval, live health, run evidence, scoped log,
+incidents, conf lookup, bug filing, code scanning. `chat` holding an empty list is a guarantee,
+not an oversight: anything added to it stops being general chat and becomes an assistant that
+sometimes invents claims about this installation.
+
+`chat` escalates to `varaverk` on its own when a question is genuinely about Varaverk, and
+reverts if the index turns out to have nothing — so the loose profile is safe to sit in.
+
+This used to live in five places — history depth in the endpoint, capabilities in `include/ai.php`,
+label and depth again in JavaScript, a prompt branch in the worker, and a label map on the
+Scheduler page. They had already drifted: the JavaScript knew three profiles where PHP knew four.
+The system prompts still live in `Tools/ai_chat_worker.php`, because they have one reader and
+moving them would relocate the most delicate text in the subsystem without removing a duplicate.
+
+## ━━━ CONVERSATIONS ━━━
+
+Chats are stored server-side under `AI_DATA_DIR/ai_chats/`, one JSON file each, saved
+automatically when a turn completes and pruned to `AI_CHAT_HISTORY_MAX` (default 10, oldest
+first by creation).
+
+There is no Save button. A conversation worth keeping is not reliably one you knew was worth
+keeping while you were having it.
+
+The same store backs the AI tab and the Monitor tab's AI row, so a thread started on the
+dashboard is the one you carry on in the tab. Messages are re-validated per message on the way
+in — a stored chat is replayed into a later prompt when reopened, so an unchecked role written
+there would be an injection that survives a reload rather than one turn.
+
+Reopened chats render as plain turns: sources, reasoning and timings describe one generation and
+are not stored, because redrawing them beside a transcript that may be continued under a
+different profile would be citing evidence for an answer no longer being made.
+
 ## ━━━ TOKEN ACCOUNTING ━━━
 
-Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai_token_history.db`):
+Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
 
 ```
 date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s
@@ -247,13 +292,13 @@ straight from the shell, since it is just a delimited file:
 
 ```bash
 # tokens used today
-awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai_token_history.db
+awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai/ai_token_history.db
 ```
 
 **The host column is where the turn ran, not where the file is read.** Each host writes only its
 own rows.
 
-`ai_token_sync.sh` pulls each partner's ledger into `/tmp/.cache/vv/ai/<slot>.tokens.db` — the
+`ai_token_sync.sh` pulls each partner's ledger into `$AI_TOKEN_CACHE_DIR/<slot>.tokens.db` (`/tmp/varaverk/ai/`) — the
 same trick `conf_sync.sh` uses for partner confs, and it runs from
 `INTERMEDIATE_MAINTENANCE_SCRIPTS` every four hours. The tab then reads every ledger it can see,
 so a fleet total is a fleet total.

Deployment/README-Deployment.md

@@ -135,6 +135,28 @@ and a third node would work with no template change at all.
 |--------|------|-------------|
 | `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` |
 | `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation |
+| `migrate_data_layout.sh` | Move everything persisted into the rooted `data/` tree | Once per host, manually. Idempotent. |
+
+### 📦  Data layout migration — `migrate_data_layout.sh`
+
+`conf_upgrade.sh` adds keys the template has and the installation does not; it never rewrites a
+value you already have. That is exactly what you want from it, and exactly why it cannot perform
+a layout migration — the paths being moved are *existing* keys, so their values would keep
+pointing at the old layout forever while the new directory variables sat beside them unused.
+
+So this rewrites those values and moves the files to match. Both halves or neither.
+
+```bash
+Deployment/migrate_data_layout.sh --dry-run    # always first
+Deployment/migrate_data_layout.sh
+```
+
+It refuses to run while a job from *this* installation is active — scoped to the installation's
+own path, because `pgrep` is system-wide and a box running both a production checkout and a
+development clone will otherwise always look busy. `--force` overrides.
+
+Each host runs it itself: `data/` is gitignored, so a restructure travels as code and conf while
+the files stay where they are. See `data/README.md` for the resulting layout.
 
 | Template | Role |
 |----------|------|

Fallback/Manual-Fallback.md

@@ -314,7 +314,7 @@ FALLBACK_HOST2_WRITEBACK_TIER1=(
 Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
 
 > In a shell where load_config.sh is not sourced, use the full path:
-> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode)
+> `/boot/config/plugins/varaverk/data/state/fallback_state.db` (internal storage mode)
 
 ```
 state=NORMAL            # NORMAL | FALLBACK | NO_INTERNET | DARK

Notes_AI-Design.md

@@ -426,7 +426,7 @@ to begin with.
 
 Treat that as a deliberate boundary, not a happy accident:
 
-- **index tracked files only** — never walk `Configurations/`, `State_Files/`, or `data/`
+- **index tracked files only** — never walk `Configurations/` or `data/`
 - a live conf value that the model genuinely needs should arrive through a *tool call* at
   query time, subject to the same redaction rules as everything else in the Security section,
   not be baked into a vector at index time

Plugin/unraid/README-unraid.md

@@ -105,11 +105,39 @@ once, and the endpoint, the cache writer, and the page all pick it up together.
 | `api/` | 50 | JSON endpoints the pages poll, plus action endpoints (run a script, stop a job, toggle a flag) |
 | `include/` | 16 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth |
 
-**Caching.** Several endpoints serve from `/tmp/vv_cache` (tmpfs) rather than hitting live
+**Caching.** Several endpoints serve from `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs) rather than hitting live
 APIs on every page view, refreshed by `Tools/api_cache_writer.sh`. `?live=1` bypasses the
 cache. A missing or unparseable cache always falls back to a live call, so the cache can never
 be the reason a page fails to load.
 
+Anything expensive belongs in that one collection rather than in each consumer. `vv_ai_stats()`
+costs about a second — mostly waiting on Ollama and `nvidia-smi` — and the AI tab was paying it
+every 30 seconds per open tab; it is now written once to the `ai` cache and read by the tab, the
+Monitor row and the dock alike. Polling faster cannot make a figure newer, it only decides how
+soon a page notices the writer's update.
+
+> `Tools/api_cache_writer.php` rebuilds the monitor payload **independently** of
+> `api/monitor.php`, and is what the page normally reads — the endpoint only assembles one on a
+> cache miss. A key added to the endpoint and not to the writer leaves its card loading forever
+> on every ordinary page load and working only on the request that happens to miss.
+
+**One widget, rendered twice.** `include/ai_chat.php` owns the conversation surface — profile
+bar, transcript, composer, source viewer, stored-chat list — and both the AI tab and the Monitor
+tab's AI row construct it. Every id is composed from a prefix so instances can coexist, and each
+tears down the previous holder of its prefix, because Unraid swaps tab content by AJAX without
+unloading the old page's JavaScript.
+
+The Scheduler tab's dock is deliberately *not* built on it: a one-line bar that follows the view
+you have open, with its own scope chip, fix flow and incident capture, is a different component
+that happens to share an endpoint. Folding it in would produce one widget with two personalities
+and a mode flag choosing between them.
+
+**Timezone.** `include/config.php` adopts Unraid's own `timeZone` from `ident.cfg` for the whole
+PHP layer. PHP here has no `date.timezone` and therefore ran in UTC while the server ran local,
+so every date this layer produced was offset from every date the shell layer wrote — and the two
+are compared constantly. `/etc/php.ini` is the wrong fix twice over: `/etc` is a RAM filesystem
+so the edit dies at reboot, and it would retimezone every other PHP application on the box.
+
 ---
 
 ## ━━━ WHAT GUARDS THE API LAYER ━━━

Plugin/unraid/Tools/README-Tools.md

@@ -23,7 +23,7 @@ containers, VMs, transcodes, arr library counts. Building that live on every pag
 dozens of API calls and `docker inspect` runs per refresh, from a WebGUI that is already the
 first thing to slow down under load.
 
-Instead, both are written to `/tmp/vv_cache` (tmpfs — RAM-speed reads, cleared on reboot) and
+Instead, both are written to `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs — RAM-speed reads, cleared on reboot) and
 the pages serve from there.
 
 ### ⚡  `api_cache_writer.sh` — local payloads, every minute
@@ -128,7 +128,7 @@ that prevents shadow directories.
                                   │
   Tools/api_cache_writer.php ─────┘   same builders → cache can't disagree with live
         │
-        └─► /tmp/vv_cache/monitor.json ──► served by api/monitor.php unless ?live=1
+        └─► /tmp/varaverk/api/monitor.json ──► served by api/monitor.php unless ?live=1
 ```
 
 Adding a metric means adding it in `include/` once. The page, the live endpoint, and the cache

README.md

@@ -618,10 +618,29 @@ varaverk/
 ├── Kernel/                  ← Kernel module config — loaded at boot by go file
 │   README: README-Kernel.md
 │
-└── Tools/                   ← Situational utilities: repair, export, emergency tools
+├── Tools/                   ← Situational utilities: repair, export, emergency tools
-    README: README-Tools.md
+│   README: README-Tools.md
+│
+└── data/                    ← Everything persisted on disk — the one on-disk root
+    README: data/README.md      gitignored except that README; per-host, never synced
+    ├── db/                  ← statistics, histories, counters, blocklists
+    ├── state/               ← runtime state: watchdogs, fallback, transcode, setup
+    ├── ai/                  ← retrieval index, memory, token ledger, bugs, saved chats
+    ├── cache/               ← persistent backups of the tmpfs caches, and only those
+    └── logs/                ← retained log output
 ```
 
+**State is data.** It used to live in a sibling `State_Files/`, with the conf-cache backup off in
+a third place and the arr backups loose in `data/`'s root — no single decision wrong, but nothing
+answered "what does Varaverk keep on disk". `STATE_DIR` still exists and still means the same
+thing; only its value moved, which is why that restructure touched none of the scripts built on
+it.
+
+**The RAM half is separate on purpose.** `VV_CACHE_ROOT` (`/tmp/varaverk/`) holds the WebGUI
+payload cache, the arr caches, the partner conf cache, partner token ledgers, and in-flight job
+files. It is read every second and rewritten by the hundred megabytes; it does not belong on
+flash. Both roots are defined once in `master.conf` and read by the shell and PHP layers alike.
+
 ---
 
 ## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━

System_Essentials/Manual-System_Essentials.md

@@ -80,7 +80,7 @@ and the continuous scripts (start_webhook_listener, fallback), which are the fir
 steps that actually touch containers.
 
 Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
-`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
+`/tmp/varaverk/conf/` is wiped on reboot. Saving it must happen before anything
 else shuts down or changes state.
 
 ---
@@ -100,10 +100,10 @@ conf_sync.sh --log             Verbose output
 ### What It Syncs
 
 - **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
-  via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
+  via SCP → writes to local `/tmp/varaverk/conf/${partner_id}.conf`
 - **Push**: sends own `Configurations/${my_id}.conf` to partner's
-  `/tmp/.cache/vv/d/${my_id}.conf` via SCP
+  `/tmp/varaverk/conf/${my_id}.conf` via SCP
-- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
+- **Own conf in local cache**: copies own conf to `/tmp/varaverk/conf/${my_id}.conf`
   on full sync (so the cache has a complete snapshot of all confs)
 
 Only partner confs are sourced from cache — `load_config.sh` always reads own conf
@@ -136,7 +136,7 @@ A notification fires if any partner fails — check partner reachability via Tai
 
 ### What It Does
 
-At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
+At array stop, copies all partner confs from `/tmp/varaverk/conf/` to
 `$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
 the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
 can't reach the partner.
@@ -164,7 +164,7 @@ conf_cache_save.sh --log        # verbose output
 ### What It Does
 
 At array start (after `conf_sync.sh`), checks which partner confs are missing from
-`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
+`/tmp/varaverk/conf/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
 if a backup exists there.
 
 Always removes the persistent backup when done — whether used or not. On a normal
@@ -434,8 +434,8 @@ REBOOT_VM_WAIT=30
 # STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
 # They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
 # (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
-STATE_DIR="${SCRIPTS_DIR}/State_Files"
+STATE_DIR="${DATA_DIR}/state"
-PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
+PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
 ```
 
 ---
@@ -446,7 +446,7 @@ PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
 
 ```bash
 # Check what's in the RAM cache:
-ls -la /tmp/.cache/vv/d/
+ls -la /tmp/varaverk/conf/
 
 # Check what's in the persistent backup:
 ls -la "$PERSISTENT_CONF_CACHE/"    # set SCRIPTS_DIR first or use full path

System_Essentials/README-System_Essentials.md

@@ -35,7 +35,7 @@ any container-connecting or continuous scripts start.
 **Partner Conf Lost Across Reboots When Partner is Down**
 Scripts like `fallback.sh` need the partner's conf vars (credentials, container
 names, tier delays) to operate. The partner conf lives in a RAM cache at
-`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
+`/tmp/varaverk/conf/` — wiped every reboot. At array start, `conf_sync.sh` pulls
 a fresh copy from the partner. But if the partner is offline at boot time, the
 pull fails and fallback has no partner vars to work with.
 
@@ -120,16 +120,16 @@ Plugin/unraid/Watchdogs/System/
 ```
 Array starts (array_started.sh, ARRAY_START_SCRIPTS):
     │
-    ├─ conf_sync.sh              ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
+    ├─ conf_sync.sh              ← SSH/SCP: pull partner confs into /tmp/varaverk/conf/
-    │                              push own conf to partner's /tmp/.cache/vv/d/
+    │                              push own conf to partner's /tmp/varaverk/conf/
     ├─ conf_cache_restore.sh     ← if partner was down: load last-known-good conf from
-    │                              $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
+    │                              $PERSISTENT_CONF_CACHE into /tmp/varaverk/conf/
     ├─ docker_syslog_filter.sh   ← before any container starts (veth filter must be live)
     └─ inotify_tuning.sh         ← before docker_network_connect.sh and continuous scripts
 
 Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
     │
-    ├─ conf_cache_save.sh        ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
+    ├─ conf_cache_save.sh        ← FIRST: snapshot /tmp/varaverk/conf/ → $PERSISTENT_CONF_CACHE
     │                              while RAM cache is still fresh
     ├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
     └─ ...other stop scripts...

Watchdogs/Manual-Watchdogs.md

@@ -521,7 +521,7 @@ at `$PERSISTENT_CONF_CACHE` while the partner is offline.
 **Remote online:** removes the persistent backup if one exists. It is not needed —
 `conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
 
-**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
+**Remote offline:** copies partner confs from the RAM cache (`$CONF_RAM_CACHE_DIR`, `/tmp/varaverk/conf/`) to
 `$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
 an extended outage. If this host reboots while the partner is still down,
 `conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
@@ -821,8 +821,8 @@ storage_watchdog.sh --status
 ### stability_watchdog Rebooted Unexpectedly
 
 ```bash
-# Check the reboot log (survives reboots — in State_Files/):
+# Check the reboot log (survives reboots — in data/state/):
-cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
+cat /boot/config/plugins/varaverk/data/state/system_watchdog_reboots.db
 # Shows timestamp and reason for each watchdog-triggered reboot
 
 # Check syslog near the reboot time:

data/README.md

@@ -0,0 +1,96 @@
+# `data/` — everything Varaverk keeps on disk
+
+This directory is the single on-disk root. If Varaverk persists something across a reboot, it is
+under here. Move `DATA_DIR` in `master.conf` and the whole tree follows.
+
+This file is the only tracked thing in here. Everything else is runtime data, gitignored, and
+per-host — none of it syncs, none of it belongs in the repo, and losing any of it costs at most
+a rebuild.
+
+---
+
+## Layout
+
+```
+data/
+├── db/       statistics, histories, counters, blocklists — things that accumulate
+├── state/    runtime state for every script: watchdogs, fallback, transcode, setup
+├── ai/       retrieval index, operator memory, token ledger, filed bugs, saved chats
+├── cache/    persistent backups of the tmpfs caches — and only those
+│   ├── arr/    *_tracked_cache.json, restored into tmpfs on demand
+│   └── conf/   partner host*.conf snapshot (0700 — holds credentials)
+└── logs/     retained log output
+```
+
+Each has a conf variable. Never hardcode a path into a script; use the variable, because the
+variable is what a storage-mode migration rewrites.
+
+| Directory | Variable | Notes |
+|---|---|---|
+| `db/` | `DB_DIR` | |
+| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
+| `ai/` | `AI_DATA_DIR` | |
+| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
+| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
+
+---
+
+## What is NOT here
+
+**The tmpfs caches.** They live under `VV_CACHE_ROOT` (`/tmp/varaverk/`) and must stay there.
+The WebGUI payload cache is read every one to five seconds by every open tab, and the arr item
+caches are rewritten by the hundred megabytes. On flash that is pointless write load for data
+whose entire purpose is to be cheap and disposable.
+
+```
+/tmp/varaverk/
+├── api/    WebGUI payloads — monitor, arrs, ai
+├── arr/    arr payloads (backed up to data/cache/arr/)
+├── conf/   partner confs   (backed up to data/cache/conf/)
+├── ai/     partner token ledgers — deliberately never backed up
+└── jobs/   in-flight AI answers and container actions (0700)
+```
+
+**Locks.** `/tmp/unraid_locks/` — deliberately outside both roots, and deliberately on tmpfs so a
+lock cannot outlive the boot that took it.
+
+**Configuration.** `Configurations/` holds the confs. Data and config stay apart: one is written
+by scripts, the other by you.
+
+---
+
+## Where a new file goes
+
+Ask what happens if it is deleted.
+
+- *Something is permanently lost* → it is a source of truth. `db/` if it accumulates, `state/`
+  if it describes right now.
+- *It gets re-fetched and nothing else changes* → it is a cache. If it also lives in tmpfs, its
+  backup goes in `cache/`. If it only lives here, it is not really a cache — put it in `db/`.
+- *Nothing at all* → it should not be written to disk in the first place.
+
+`cache/` is the one that gets misused. It means "persistent backup of a tmpfs cache", not
+"anything cache-shaped". A file that is only ever here is a source of truth no matter what it is
+called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
+reason.
+
+---
+
+## History
+
+Until 2026-08-08 this was two roots and two strays: `data/` and `State_Files/` as siblings, the
+conf-cache backup off in `SCRIPTS_DIR/.cache/vv/d`, and the arr backups loose in `data/`'s root.
+No single decision there was wrong. Together they meant nothing answered "what does Varaverk keep
+on disk", and the PHP layer — which cannot source bash — restated the paths it needed, so the two
+layers agreed only by hand.
+
+State is data. It is the data that happens to describe right now, so it belongs under the same
+root as the rest.
+
+`STATE_DIR` kept its name and changed only its value, which is why that restructure did not touch
+the 15 conf entries, 18 shell paths and 23 PHP paths built on it.
+
+**Migrating an installation:** `Deployment/migrate_data_layout.sh --dry-run` first, then without
+the flag. It is idempotent, it moves rather than copies, and it rewrites the conf values that
+`conf_upgrade` deliberately will not touch. Each host runs it itself — this directory is
+gitignored, so the restructure travels as code and conf while the files stay put.