diff --git a/.gitignore b/.gitignore index 709cd97..3482954 100644 --- a/.gitignore +++ b/.gitignore @@ -10,7 +10,11 @@ Configurations/*.bak Notes_To-Do.md # ── Runtime state, data, logs ───────────────────────────────────────────────── -data/ +# Contents, not the directory itself. Ignoring "data/" outright means git never descends into +# it, and a negation for a file inside an excluded directory is silently ineffective — so the +# README explaining what data/ is would be the one file missing from every installation of it. +data/* +!data/README.md State_Files/ .cache/ *.log diff --git a/AI/README-AI.md b/AI/README-AI.md index 6e4aec6..465e4e0 100644 --- a/AI/README-AI.md +++ b/AI/README-AI.md @@ -119,7 +119,7 @@ questions this answers, intent retrieves better and costs far less. **Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model. -`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential +`Configurations/` and `data/` are gitignored, so every file holding a credential was never in the repo to begin with. The index therefore describes the full conf schema (via the tracked templates, which carry all the explanatory comments) while structurally **never containing a secret**. @@ -228,9 +228,54 @@ and a pull that changed twelve files costs a few seconds. --- +## ━━━ PROFILES ━━━ + +A profile is a contract plus a set of inputs. `Plugin/unraid/include/ai_profiles.php` is the one +definition of both, read by the endpoint, the worker, the shared chat include and the Scheduler +dock. + +| Profile | Turns | Retrieves | Notes | +|---|---|---|---| +| `varaverk` | 3 | yes | answers only from the index, with citations. The default. | +| `chat` | 8 | **no** | ordinary conversation. Holds zero capabilities, deliberately. | +| `code` | 4 | no | drafts shell for Custom Scripts; scans its own output for destructive ops | +| `troubleshoot` | 2 | yes | reasons from an open log first, docs second. May file bug reports. | + +Capabilities are granted per profile — retrieval, live health, run evidence, scoped log, +incidents, conf lookup, bug filing, code scanning. `chat` holding an empty list is a guarantee, +not an oversight: anything added to it stops being general chat and becomes an assistant that +sometimes invents claims about this installation. + +`chat` escalates to `varaverk` on its own when a question is genuinely about Varaverk, and +reverts if the index turns out to have nothing — so the loose profile is safe to sit in. + +This used to live in five places — history depth in the endpoint, capabilities in `include/ai.php`, +label and depth again in JavaScript, a prompt branch in the worker, and a label map on the +Scheduler page. They had already drifted: the JavaScript knew three profiles where PHP knew four. +The system prompts still live in `Tools/ai_chat_worker.php`, because they have one reader and +moving them would relocate the most delicate text in the subsystem without removing a duplicate. + +## ━━━ CONVERSATIONS ━━━ + +Chats are stored server-side under `AI_DATA_DIR/ai_chats/`, one JSON file each, saved +automatically when a turn completes and pruned to `AI_CHAT_HISTORY_MAX` (default 10, oldest +first by creation). + +There is no Save button. A conversation worth keeping is not reliably one you knew was worth +keeping while you were having it. + +The same store backs the AI tab and the Monitor tab's AI row, so a thread started on the +dashboard is the one you carry on in the tab. Messages are re-validated per message on the way +in — a stored chat is replayed into a later prompt when reopened, so an unchecked role written +there would be an injection that survives a reload rather than one turn. + +Reopened chats render as plain turns: sources, reasoning and timings describe one generation and +are not stored, because redrawing them beside a transcript that may be continued under a +different profile would be citing evidence for an answer no longer being made. + ## ━━━ TOKEN ACCOUNTING ━━━ -Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai_token_history.db`): +Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`): ``` date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s @@ -247,13 +292,13 @@ straight from the shell, since it is just a delimited file: ```bash # tokens used today -awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai_token_history.db +awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai/ai_token_history.db ``` **The host column is where the turn ran, not where the file is read.** Each host writes only its own rows. -`ai_token_sync.sh` pulls each partner's ledger into `/tmp/.cache/vv/ai/.tokens.db` — the +`ai_token_sync.sh` pulls each partner's ledger into `$AI_TOKEN_CACHE_DIR/.tokens.db` (`/tmp/varaverk/ai/`) — the same trick `conf_sync.sh` uses for partner confs, and it runs from `INTERMEDIATE_MAINTENANCE_SCRIPTS` every four hours. The tab then reads every ledger it can see, so a fleet total is a fleet total. diff --git a/Deployment/README-Deployment.md b/Deployment/README-Deployment.md index f13f0d8..5cd0dad 100644 --- a/Deployment/README-Deployment.md +++ b/Deployment/README-Deployment.md @@ -135,6 +135,28 @@ and a third node would work with no template change at all. |--------|------|-------------| | `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` | | `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation | +| `migrate_data_layout.sh` | Move everything persisted into the rooted `data/` tree | Once per host, manually. Idempotent. | + +### 📦 Data layout migration — `migrate_data_layout.sh` + +`conf_upgrade.sh` adds keys the template has and the installation does not; it never rewrites a +value you already have. That is exactly what you want from it, and exactly why it cannot perform +a layout migration — the paths being moved are *existing* keys, so their values would keep +pointing at the old layout forever while the new directory variables sat beside them unused. + +So this rewrites those values and moves the files to match. Both halves or neither. + +```bash +Deployment/migrate_data_layout.sh --dry-run # always first +Deployment/migrate_data_layout.sh +``` + +It refuses to run while a job from *this* installation is active — scoped to the installation's +own path, because `pgrep` is system-wide and a box running both a production checkout and a +development clone will otherwise always look busy. `--force` overrides. + +Each host runs it itself: `data/` is gitignored, so a restructure travels as code and conf while +the files stay where they are. See `data/README.md` for the resulting layout. | Template | Role | |----------|------| diff --git a/Fallback/Manual-Fallback.md b/Fallback/Manual-Fallback.md index 691e1d7..1e51c23 100644 --- a/Fallback/Manual-Fallback.md +++ b/Fallback/Manual-Fallback.md @@ -314,7 +314,7 @@ FALLBACK_HOST2_WRITEBACK_TIER1=( Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata) > In a shell where load_config.sh is not sourced, use the full path: -> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode) +> `/boot/config/plugins/varaverk/data/state/fallback_state.db` (internal storage mode) ``` state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK diff --git a/Notes_AI-Design.md b/Notes_AI-Design.md index e885238..be48062 100644 --- a/Notes_AI-Design.md +++ b/Notes_AI-Design.md @@ -426,7 +426,7 @@ to begin with. Treat that as a deliberate boundary, not a happy accident: -- **index tracked files only** — never walk `Configurations/`, `State_Files/`, or `data/` +- **index tracked files only** — never walk `Configurations/` or `data/` - a live conf value that the model genuinely needs should arrive through a *tool call* at query time, subject to the same redaction rules as everything else in the Security section, not be baked into a vector at index time diff --git a/Plugin/unraid/README-unraid.md b/Plugin/unraid/README-unraid.md index edae6b1..602b07a 100644 --- a/Plugin/unraid/README-unraid.md +++ b/Plugin/unraid/README-unraid.md @@ -105,11 +105,39 @@ once, and the endpoint, the cache writer, and the page all pick it up together. | `api/` | 50 | JSON endpoints the pages poll, plus action endpoints (run a script, stop a job, toggle a flag) | | `include/` | 16 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth | -**Caching.** Several endpoints serve from `/tmp/vv_cache` (tmpfs) rather than hitting live +**Caching.** Several endpoints serve from `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs) rather than hitting live APIs on every page view, refreshed by `Tools/api_cache_writer.sh`. `?live=1` bypasses the cache. A missing or unparseable cache always falls back to a live call, so the cache can never be the reason a page fails to load. +Anything expensive belongs in that one collection rather than in each consumer. `vv_ai_stats()` +costs about a second — mostly waiting on Ollama and `nvidia-smi` — and the AI tab was paying it +every 30 seconds per open tab; it is now written once to the `ai` cache and read by the tab, the +Monitor row and the dock alike. Polling faster cannot make a figure newer, it only decides how +soon a page notices the writer's update. + +> `Tools/api_cache_writer.php` rebuilds the monitor payload **independently** of +> `api/monitor.php`, and is what the page normally reads — the endpoint only assembles one on a +> cache miss. A key added to the endpoint and not to the writer leaves its card loading forever +> on every ordinary page load and working only on the request that happens to miss. + +**One widget, rendered twice.** `include/ai_chat.php` owns the conversation surface — profile +bar, transcript, composer, source viewer, stored-chat list — and both the AI tab and the Monitor +tab's AI row construct it. Every id is composed from a prefix so instances can coexist, and each +tears down the previous holder of its prefix, because Unraid swaps tab content by AJAX without +unloading the old page's JavaScript. + +The Scheduler tab's dock is deliberately *not* built on it: a one-line bar that follows the view +you have open, with its own scope chip, fix flow and incident capture, is a different component +that happens to share an endpoint. Folding it in would produce one widget with two personalities +and a mode flag choosing between them. + +**Timezone.** `include/config.php` adopts Unraid's own `timeZone` from `ident.cfg` for the whole +PHP layer. PHP here has no `date.timezone` and therefore ran in UTC while the server ran local, +so every date this layer produced was offset from every date the shell layer wrote — and the two +are compared constantly. `/etc/php.ini` is the wrong fix twice over: `/etc` is a RAM filesystem +so the edit dies at reboot, and it would retimezone every other PHP application on the box. + --- ## ━━━ WHAT GUARDS THE API LAYER ━━━ diff --git a/Plugin/unraid/Tools/README-Tools.md b/Plugin/unraid/Tools/README-Tools.md index e850f50..435a2a9 100644 --- a/Plugin/unraid/Tools/README-Tools.md +++ b/Plugin/unraid/Tools/README-Tools.md @@ -23,7 +23,7 @@ containers, VMs, transcodes, arr library counts. Building that live on every pag dozens of API calls and `docker inspect` runs per refresh, from a WebGUI that is already the first thing to slow down under load. -Instead, both are written to `/tmp/vv_cache` (tmpfs — RAM-speed reads, cleared on reboot) and +Instead, both are written to `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs — RAM-speed reads, cleared on reboot) and the pages serve from there. ### ⚡ `api_cache_writer.sh` — local payloads, every minute @@ -128,7 +128,7 @@ that prevents shadow directories. │ Tools/api_cache_writer.php ─────┘ same builders → cache can't disagree with live │ - └─► /tmp/vv_cache/monitor.json ──► served by api/monitor.php unless ?live=1 + └─► /tmp/varaverk/api/monitor.json ──► served by api/monitor.php unless ?live=1 ``` Adding a metric means adding it in `include/` once. The page, the live endpoint, and the cache diff --git a/README.md b/README.md index 99f9a3b..eddef26 100644 --- a/README.md +++ b/README.md @@ -618,10 +618,29 @@ varaverk/ ├── Kernel/ ← Kernel module config — loaded at boot by go file │ README: README-Kernel.md │ -└── Tools/ ← Situational utilities: repair, export, emergency tools - README: README-Tools.md +├── Tools/ ← Situational utilities: repair, export, emergency tools +│ README: README-Tools.md +│ +└── data/ ← Everything persisted on disk — the one on-disk root + README: data/README.md gitignored except that README; per-host, never synced + ├── db/ ← statistics, histories, counters, blocklists + ├── state/ ← runtime state: watchdogs, fallback, transcode, setup + ├── ai/ ← retrieval index, memory, token ledger, bugs, saved chats + ├── cache/ ← persistent backups of the tmpfs caches, and only those + └── logs/ ← retained log output ``` +**State is data.** It used to live in a sibling `State_Files/`, with the conf-cache backup off in +a third place and the arr backups loose in `data/`'s root — no single decision wrong, but nothing +answered "what does Varaverk keep on disk". `STATE_DIR` still exists and still means the same +thing; only its value moved, which is why that restructure touched none of the scripts built on +it. + +**The RAM half is separate on purpose.** `VV_CACHE_ROOT` (`/tmp/varaverk/`) holds the WebGUI +payload cache, the arr caches, the partner conf cache, partner token ledgers, and in-flight job +files. It is read every second and rewritten by the hundred megabytes; it does not belong on +flash. Both roots are defined once in `master.conf` and read by the shell and PHP layers alike. + --- ## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━ diff --git a/System_Essentials/Manual-System_Essentials.md b/System_Essentials/Manual-System_Essentials.md index 26ae390..7307d60 100644 --- a/System_Essentials/Manual-System_Essentials.md +++ b/System_Essentials/Manual-System_Essentials.md @@ -80,7 +80,7 @@ and the continuous scripts (start_webhook_listener, fallback), which are the fir steps that actually touch containers. Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at -`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything +`/tmp/varaverk/conf/` is wiped on reboot. Saving it must happen before anything else shuts down or changes state. --- @@ -100,10 +100,10 @@ conf_sync.sh --log Verbose output ### What It Syncs - **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk - via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf` + via SCP → writes to local `/tmp/varaverk/conf/${partner_id}.conf` - **Push**: sends own `Configurations/${my_id}.conf` to partner's - `/tmp/.cache/vv/d/${my_id}.conf` via SCP -- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf` + `/tmp/varaverk/conf/${my_id}.conf` via SCP +- **Own conf in local cache**: copies own conf to `/tmp/varaverk/conf/${my_id}.conf` on full sync (so the cache has a complete snapshot of all confs) Only partner confs are sourced from cache — `load_config.sh` always reads own conf @@ -136,7 +136,7 @@ A notification fires if any partner fails — check partner reachability via Tai ### What It Does -At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to +At array stop, copies all partner confs from `/tmp/varaverk/conf/` to `$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives the reboot and is used by `conf_cache_restore.sh` at next array start if the sync can't reach the partner. @@ -164,7 +164,7 @@ conf_cache_save.sh --log # verbose output ### What It Does At array start (after `conf_sync.sh`), checks which partner confs are missing from -`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE` +`/tmp/varaverk/conf/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE` if a backup exists there. Always removes the persistent backup when done — whether used or not. On a normal @@ -434,8 +434,8 @@ REBOOT_VM_WAIT=30 # STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf. # They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode # (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths. -STATE_DIR="${SCRIPTS_DIR}/State_Files" -PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d" +STATE_DIR="${DATA_DIR}/state" +PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf" ``` --- @@ -446,7 +446,7 @@ PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d" ```bash # Check what's in the RAM cache: -ls -la /tmp/.cache/vv/d/ +ls -la /tmp/varaverk/conf/ # Check what's in the persistent backup: ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path diff --git a/System_Essentials/README-System_Essentials.md b/System_Essentials/README-System_Essentials.md index 78fd633..29a773b 100644 --- a/System_Essentials/README-System_Essentials.md +++ b/System_Essentials/README-System_Essentials.md @@ -35,7 +35,7 @@ any container-connecting or continuous scripts start. **Partner Conf Lost Across Reboots When Partner is Down** Scripts like `fallback.sh` need the partner's conf vars (credentials, container names, tier delays) to operate. The partner conf lives in a RAM cache at -`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls +`/tmp/varaverk/conf/` — wiped every reboot. At array start, `conf_sync.sh` pulls a fresh copy from the partner. But if the partner is offline at boot time, the pull fails and fallback has no partner vars to work with. @@ -120,16 +120,16 @@ Plugin/unraid/Watchdogs/System/ ``` Array starts (array_started.sh, ARRAY_START_SCRIPTS): │ - ├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/ - │ push own conf to partner's /tmp/.cache/vv/d/ + ├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/varaverk/conf/ + │ push own conf to partner's /tmp/varaverk/conf/ ├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from - │ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/ + │ $PERSISTENT_CONF_CACHE into /tmp/varaverk/conf/ ├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live) └─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS): │ - ├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE + ├─ conf_cache_save.sh ← FIRST: snapshot /tmp/varaverk/conf/ → $PERSISTENT_CONF_CACHE │ while RAM cache is still fresh ├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery └─ ...other stop scripts... diff --git a/Watchdogs/Manual-Watchdogs.md b/Watchdogs/Manual-Watchdogs.md index b6f4ba0..2eeaac3 100644 --- a/Watchdogs/Manual-Watchdogs.md +++ b/Watchdogs/Manual-Watchdogs.md @@ -521,7 +521,7 @@ at `$PERSISTENT_CONF_CACHE` while the partner is offline. **Remote online:** removes the persistent backup if one exists. It is not needed — `conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent. -**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to +**Remote offline:** copies partner confs from the RAM cache (`$CONF_RAM_CACHE_DIR`, `/tmp/varaverk/conf/`) to `$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout an extended outage. If this host reboots while the partner is still down, `conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have @@ -821,8 +821,8 @@ storage_watchdog.sh --status ### stability_watchdog Rebooted Unexpectedly ```bash -# Check the reboot log (survives reboots — in State_Files/): -cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db +# Check the reboot log (survives reboots — in data/state/): +cat /boot/config/plugins/varaverk/data/state/system_watchdog_reboots.db # Shows timestamp and reason for each watchdog-triggered reboot # Check syslog near the reboot time: diff --git a/data/README.md b/data/README.md new file mode 100644 index 0000000..b3b5342 --- /dev/null +++ b/data/README.md @@ -0,0 +1,96 @@ +# `data/` — everything Varaverk keeps on disk + +This directory is the single on-disk root. If Varaverk persists something across a reboot, it is +under here. Move `DATA_DIR` in `master.conf` and the whole tree follows. + +This file is the only tracked thing in here. Everything else is runtime data, gitignored, and +per-host — none of it syncs, none of it belongs in the repo, and losing any of it costs at most +a rebuild. + +--- + +## Layout + +``` +data/ +├── db/ statistics, histories, counters, blocklists — things that accumulate +├── state/ runtime state for every script: watchdogs, fallback, transcode, setup +├── ai/ retrieval index, operator memory, token ledger, filed bugs, saved chats +├── cache/ persistent backups of the tmpfs caches — and only those +│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand +│ └── conf/ partner host*.conf snapshot (0700 — holds credentials) +└── logs/ retained log output +``` + +Each has a conf variable. Never hardcode a path into a script; use the variable, because the +variable is what a storage-mode migration rewrites. + +| Directory | Variable | Notes | +|---|---|---| +| `db/` | `DB_DIR` | | +| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. | +| `ai/` | `AI_DATA_DIR` | | +| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it | +| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) | + +--- + +## What is NOT here + +**The tmpfs caches.** They live under `VV_CACHE_ROOT` (`/tmp/varaverk/`) and must stay there. +The WebGUI payload cache is read every one to five seconds by every open tab, and the arr item +caches are rewritten by the hundred megabytes. On flash that is pointless write load for data +whose entire purpose is to be cheap and disposable. + +``` +/tmp/varaverk/ +├── api/ WebGUI payloads — monitor, arrs, ai +├── arr/ arr payloads (backed up to data/cache/arr/) +├── conf/ partner confs (backed up to data/cache/conf/) +├── ai/ partner token ledgers — deliberately never backed up +└── jobs/ in-flight AI answers and container actions (0700) +``` + +**Locks.** `/tmp/unraid_locks/` — deliberately outside both roots, and deliberately on tmpfs so a +lock cannot outlive the boot that took it. + +**Configuration.** `Configurations/` holds the confs. Data and config stay apart: one is written +by scripts, the other by you. + +--- + +## Where a new file goes + +Ask what happens if it is deleted. + +- *Something is permanently lost* → it is a source of truth. `db/` if it accumulates, `state/` + if it describes right now. +- *It gets re-fetched and nothing else changes* → it is a cache. If it also lives in tmpfs, its + backup goes in `cache/`. If it only lives here, it is not really a cache — put it in `db/`. +- *Nothing at all* → it should not be written to disk in the first place. + +`cache/` is the one that gets misused. It means "persistent backup of a tmpfs cache", not +"anything cache-shaped". A file that is only ever here is a source of truth no matter what it is +called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that +reason. + +--- + +## History + +Until 2026-08-08 this was two roots and two strays: `data/` and `State_Files/` as siblings, the +conf-cache backup off in `SCRIPTS_DIR/.cache/vv/d`, and the arr backups loose in `data/`'s root. +No single decision there was wrong. Together they meant nothing answered "what does Varaverk keep +on disk", and the PHP layer — which cannot source bash — restated the paths it needed, so the two +layers agreed only by hand. + +State is data. It is the data that happens to describe right now, so it belongs under the same +root as the rest. + +`STATE_DIR` kept its name and changed only its value, which is why that restructure did not touch +the 15 conf entries, 18 shell paths and 23 PHP paths built on it. + +**Migrating an installation:** `Deployment/migrate_data_layout.sh --dry-run` first, then without +the flag. It is idempotent, it moves rather than copies, and it rewrites the conf values that +`conf_upgrade` deliberately will not touch. Each host runs it itself — this directory is +gitignored, so the restructure travels as code and conf while the files stay put.