Document today's restructures, and give data/ a README that ships with it
This commit is contained in:
Gmer4Lfe
+5
-1
@@ -10,7 +10,11 @@ Configurations/*.bak
|
||||
Notes_To-Do.md
|
||||
|
||||
# ── Runtime state, data, logs ─────────────────────────────────────────────────
|
||||
data/
|
||||
# Contents, not the directory itself. Ignoring "data/" outright means git never descends into
|
||||
# it, and a negation for a file inside an excluded directory is silently ineffective — so the
|
||||
# README explaining what data/ is would be the one file missing from every installation of it.
|
||||
data/*
|
||||
!data/README.md
|
||||
State_Files/
|
||||
.cache/
|
||||
*.log
|
||||
|
||||
+49
-4
@@ -119,7 +119,7 @@ questions this answers, intent retrieves better and costs far less.
|
||||
|
||||
**Only `git ls-files` is ever indexed.** This is not a convenience — it is the security model.
|
||||
|
||||
`Configurations/`, `State_Files/` and `data/` are gitignored, so every file holding a credential
|
||||
`Configurations/` and `data/` are gitignored, so every file holding a credential
|
||||
was never in the repo to begin with. The index therefore describes the full conf schema (via the
|
||||
tracked templates, which carry all the explanatory comments) while structurally **never
|
||||
containing a secret**.
|
||||
@@ -228,9 +228,54 @@ and a pull that changed twelve files costs a few seconds.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ PROFILES ━━━
|
||||
|
||||
A profile is a contract plus a set of inputs. `Plugin/unraid/include/ai_profiles.php` is the one
|
||||
definition of both, read by the endpoint, the worker, the shared chat include and the Scheduler
|
||||
dock.
|
||||
|
||||
| Profile | Turns | Retrieves | Notes |
|
||||
|---|---|---|---|
|
||||
| `varaverk` | 3 | yes | answers only from the index, with citations. The default. |
|
||||
| `chat` | 8 | **no** | ordinary conversation. Holds zero capabilities, deliberately. |
|
||||
| `code` | 4 | no | drafts shell for Custom Scripts; scans its own output for destructive ops |
|
||||
| `troubleshoot` | 2 | yes | reasons from an open log first, docs second. May file bug reports. |
|
||||
|
||||
Capabilities are granted per profile — retrieval, live health, run evidence, scoped log,
|
||||
incidents, conf lookup, bug filing, code scanning. `chat` holding an empty list is a guarantee,
|
||||
not an oversight: anything added to it stops being general chat and becomes an assistant that
|
||||
sometimes invents claims about this installation.
|
||||
|
||||
`chat` escalates to `varaverk` on its own when a question is genuinely about Varaverk, and
|
||||
reverts if the index turns out to have nothing — so the loose profile is safe to sit in.
|
||||
|
||||
This used to live in five places — history depth in the endpoint, capabilities in `include/ai.php`,
|
||||
label and depth again in JavaScript, a prompt branch in the worker, and a label map on the
|
||||
Scheduler page. They had already drifted: the JavaScript knew three profiles where PHP knew four.
|
||||
The system prompts still live in `Tools/ai_chat_worker.php`, because they have one reader and
|
||||
moving them would relocate the most delicate text in the subsystem without removing a duplicate.
|
||||
|
||||
## ━━━ CONVERSATIONS ━━━
|
||||
|
||||
Chats are stored server-side under `AI_DATA_DIR/ai_chats/`, one JSON file each, saved
|
||||
automatically when a turn completes and pruned to `AI_CHAT_HISTORY_MAX` (default 10, oldest
|
||||
first by creation).
|
||||
|
||||
There is no Save button. A conversation worth keeping is not reliably one you knew was worth
|
||||
keeping while you were having it.
|
||||
|
||||
The same store backs the AI tab and the Monitor tab's AI row, so a thread started on the
|
||||
dashboard is the one you carry on in the tab. Messages are re-validated per message on the way
|
||||
in — a stored chat is replayed into a later prompt when reopened, so an unchecked role written
|
||||
there would be an injection that survives a reload rather than one turn.
|
||||
|
||||
Reopened chats render as plain turns: sources, reasoning and timings describe one generation and
|
||||
are not stored, because redrawing them beside a transcript that may be continued under a
|
||||
different profile would be citing evidence for an answer no longer being made.
|
||||
|
||||
## ━━━ TOKEN ACCOUNTING ━━━
|
||||
|
||||
Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai_token_history.db`):
|
||||
Every completed `ask` appends one row to `AI_TOKEN_DB` (`data/ai/ai_token_history.db`):
|
||||
|
||||
```
|
||||
date|time|host|profile|source|prompt_tokens|completion_tokens|tok_s
|
||||
@@ -247,13 +292,13 @@ straight from the shell, since it is just a delimited file:
|
||||
|
||||
```bash
|
||||
# tokens used today
|
||||
awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai_token_history.db
|
||||
awk -F'|' -v d="$(date +%F)" '$1==d {p+=$6; c+=$7} END {print p+c}' data/ai/ai_token_history.db
|
||||
```
|
||||
|
||||
**The host column is where the turn ran, not where the file is read.** Each host writes only its
|
||||
own rows.
|
||||
|
||||
`ai_token_sync.sh` pulls each partner's ledger into `/tmp/.cache/vv/ai/<slot>.tokens.db` — the
|
||||
`ai_token_sync.sh` pulls each partner's ledger into `$AI_TOKEN_CACHE_DIR/<slot>.tokens.db` (`/tmp/varaverk/ai/`) — the
|
||||
same trick `conf_sync.sh` uses for partner confs, and it runs from
|
||||
`INTERMEDIATE_MAINTENANCE_SCRIPTS` every four hours. The tab then reads every ledger it can see,
|
||||
so a fleet total is a fleet total.
|
||||
|
||||
@@ -135,6 +135,28 @@ and a third node would work with no template change at all.
|
||||
|--------|------|-------------|
|
||||
| `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` |
|
||||
| `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation |
|
||||
| `migrate_data_layout.sh` | Move everything persisted into the rooted `data/` tree | Once per host, manually. Idempotent. |
|
||||
|
||||
### 📦 Data layout migration — `migrate_data_layout.sh`
|
||||
|
||||
`conf_upgrade.sh` adds keys the template has and the installation does not; it never rewrites a
|
||||
value you already have. That is exactly what you want from it, and exactly why it cannot perform
|
||||
a layout migration — the paths being moved are *existing* keys, so their values would keep
|
||||
pointing at the old layout forever while the new directory variables sat beside them unused.
|
||||
|
||||
So this rewrites those values and moves the files to match. Both halves or neither.
|
||||
|
||||
```bash
|
||||
Deployment/migrate_data_layout.sh --dry-run # always first
|
||||
Deployment/migrate_data_layout.sh
|
||||
```
|
||||
|
||||
It refuses to run while a job from *this* installation is active — scoped to the installation's
|
||||
own path, because `pgrep` is system-wide and a box running both a production checkout and a
|
||||
development clone will otherwise always look busy. `--force` overrides.
|
||||
|
||||
Each host runs it itself: `data/` is gitignored, so a restructure travels as code and conf while
|
||||
the files stay where they are. See `data/README.md` for the resulting layout.
|
||||
|
||||
| Template | Role |
|
||||
|----------|------|
|
||||
|
||||
@@ -314,7 +314,7 @@ FALLBACK_HOST2_WRITEBACK_TIER1=(
|
||||
Location: `$STATE_DIR/fallback_state.db` (survives reboots — boot device or appdata)
|
||||
|
||||
> In a shell where load_config.sh is not sourced, use the full path:
|
||||
> `/boot/config/plugins/varaverk/State_Files/fallback_state.db` (internal storage mode)
|
||||
> `/boot/config/plugins/varaverk/data/state/fallback_state.db` (internal storage mode)
|
||||
|
||||
```
|
||||
state=NORMAL # NORMAL | FALLBACK | NO_INTERNET | DARK
|
||||
|
||||
+1
-1
@@ -426,7 +426,7 @@ to begin with.
|
||||
|
||||
Treat that as a deliberate boundary, not a happy accident:
|
||||
|
||||
- **index tracked files only** — never walk `Configurations/`, `State_Files/`, or `data/`
|
||||
- **index tracked files only** — never walk `Configurations/` or `data/`
|
||||
- a live conf value that the model genuinely needs should arrive through a *tool call* at
|
||||
query time, subject to the same redaction rules as everything else in the Security section,
|
||||
not be baked into a vector at index time
|
||||
|
||||
@@ -105,11 +105,39 @@ once, and the endpoint, the cache writer, and the page all pick it up together.
|
||||
| `api/` | 50 | JSON endpoints the pages poll, plus action endpoints (run a script, stop a job, toggle a flag) |
|
||||
| `include/` | 16 | Shared builders and helpers — `vv_monitor_*`, `vv_arrs_*`, `vv_docker_*`, config read/write, auth |
|
||||
|
||||
**Caching.** Several endpoints serve from `/tmp/vv_cache` (tmpfs) rather than hitting live
|
||||
**Caching.** Several endpoints serve from `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs) rather than hitting live
|
||||
APIs on every page view, refreshed by `Tools/api_cache_writer.sh`. `?live=1` bypasses the
|
||||
cache. A missing or unparseable cache always falls back to a live call, so the cache can never
|
||||
be the reason a page fails to load.
|
||||
|
||||
Anything expensive belongs in that one collection rather than in each consumer. `vv_ai_stats()`
|
||||
costs about a second — mostly waiting on Ollama and `nvidia-smi` — and the AI tab was paying it
|
||||
every 30 seconds per open tab; it is now written once to the `ai` cache and read by the tab, the
|
||||
Monitor row and the dock alike. Polling faster cannot make a figure newer, it only decides how
|
||||
soon a page notices the writer's update.
|
||||
|
||||
> `Tools/api_cache_writer.php` rebuilds the monitor payload **independently** of
|
||||
> `api/monitor.php`, and is what the page normally reads — the endpoint only assembles one on a
|
||||
> cache miss. A key added to the endpoint and not to the writer leaves its card loading forever
|
||||
> on every ordinary page load and working only on the request that happens to miss.
|
||||
|
||||
**One widget, rendered twice.** `include/ai_chat.php` owns the conversation surface — profile
|
||||
bar, transcript, composer, source viewer, stored-chat list — and both the AI tab and the Monitor
|
||||
tab's AI row construct it. Every id is composed from a prefix so instances can coexist, and each
|
||||
tears down the previous holder of its prefix, because Unraid swaps tab content by AJAX without
|
||||
unloading the old page's JavaScript.
|
||||
|
||||
The Scheduler tab's dock is deliberately *not* built on it: a one-line bar that follows the view
|
||||
you have open, with its own scope chip, fix flow and incident capture, is a different component
|
||||
that happens to share an endpoint. Folding it in would produce one widget with two personalities
|
||||
and a mode flag choosing between them.
|
||||
|
||||
**Timezone.** `include/config.php` adopts Unraid's own `timeZone` from `ident.cfg` for the whole
|
||||
PHP layer. PHP here has no `date.timezone` and therefore ran in UTC while the server ran local,
|
||||
so every date this layer produced was offset from every date the shell layer wrote — and the two
|
||||
are compared constantly. `/etc/php.ini` is the wrong fix twice over: `/etc` is a RAM filesystem
|
||||
so the edit dies at reboot, and it would retimezone every other PHP application on the box.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT GUARDS THE API LAYER ━━━
|
||||
|
||||
@@ -23,7 +23,7 @@ containers, VMs, transcodes, arr library counts. Building that live on every pag
|
||||
dozens of API calls and `docker inspect` runs per refresh, from a WebGUI that is already the
|
||||
first thing to slow down under load.
|
||||
|
||||
Instead, both are written to `/tmp/vv_cache` (tmpfs — RAM-speed reads, cleared on reboot) and
|
||||
Instead, both are written to `$VV_CACHE_DIR` (`/tmp/varaverk/api`, tmpfs — RAM-speed reads, cleared on reboot) and
|
||||
the pages serve from there.
|
||||
|
||||
### ⚡ `api_cache_writer.sh` — local payloads, every minute
|
||||
@@ -128,7 +128,7 @@ that prevents shadow directories.
|
||||
│
|
||||
Tools/api_cache_writer.php ─────┘ same builders → cache can't disagree with live
|
||||
│
|
||||
└─► /tmp/vv_cache/monitor.json ──► served by api/monitor.php unless ?live=1
|
||||
└─► /tmp/varaverk/api/monitor.json ──► served by api/monitor.php unless ?live=1
|
||||
```
|
||||
|
||||
Adding a metric means adding it in `include/` once. The page, the live endpoint, and the cache
|
||||
|
||||
@@ -618,10 +618,29 @@ varaverk/
|
||||
├── Kernel/ ← Kernel module config — loaded at boot by go file
|
||||
│ README: README-Kernel.md
|
||||
│
|
||||
└── Tools/ ← Situational utilities: repair, export, emergency tools
|
||||
README: README-Tools.md
|
||||
├── Tools/ ← Situational utilities: repair, export, emergency tools
|
||||
│ README: README-Tools.md
|
||||
│
|
||||
└── data/ ← Everything persisted on disk — the one on-disk root
|
||||
README: data/README.md gitignored except that README; per-host, never synced
|
||||
├── db/ ← statistics, histories, counters, blocklists
|
||||
├── state/ ← runtime state: watchdogs, fallback, transcode, setup
|
||||
├── ai/ ← retrieval index, memory, token ledger, bugs, saved chats
|
||||
├── cache/ ← persistent backups of the tmpfs caches, and only those
|
||||
└── logs/ ← retained log output
|
||||
```
|
||||
|
||||
**State is data.** It used to live in a sibling `State_Files/`, with the conf-cache backup off in
|
||||
a third place and the arr backups loose in `data/`'s root — no single decision wrong, but nothing
|
||||
answered "what does Varaverk keep on disk". `STATE_DIR` still exists and still means the same
|
||||
thing; only its value moved, which is why that restructure touched none of the scripts built on
|
||||
it.
|
||||
|
||||
**The RAM half is separate on purpose.** `VV_CACHE_ROOT` (`/tmp/varaverk/`) holds the WebGUI
|
||||
payload cache, the arr caches, the partner conf cache, partner token ledgers, and in-flight job
|
||||
files. It is read every second and rewritten by the hundred megabytes; it does not belong on
|
||||
flash. Both roots are defined once in `master.conf` and read by the shell and PHP layers alike.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONFIGURATION — Configurations/ FOLDER, THREE FILES ━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
@@ -80,7 +80,7 @@ and the continuous scripts (start_webhook_listener, fallback), which are the fir
|
||||
steps that actually touch containers.
|
||||
|
||||
Why conf_cache_save is FIRST in ARRAY_STOP_SCRIPTS: the RAM cache at
|
||||
`/tmp/.cache/vv/d/` is wiped on reboot. Saving it must happen before anything
|
||||
`/tmp/varaverk/conf/` is wiped on reboot. Saving it must happen before anything
|
||||
else shuts down or changes state.
|
||||
|
||||
---
|
||||
@@ -100,10 +100,10 @@ conf_sync.sh --log Verbose output
|
||||
### What It Syncs
|
||||
|
||||
- **Pull**: reads the partner's `Configurations/${partner_id}.conf` from their disk
|
||||
via SCP → writes to local `/tmp/.cache/vv/d/${partner_id}.conf`
|
||||
via SCP → writes to local `/tmp/varaverk/conf/${partner_id}.conf`
|
||||
- **Push**: sends own `Configurations/${my_id}.conf` to partner's
|
||||
`/tmp/.cache/vv/d/${my_id}.conf` via SCP
|
||||
- **Own conf in local cache**: copies own conf to `/tmp/.cache/vv/d/${my_id}.conf`
|
||||
`/tmp/varaverk/conf/${my_id}.conf` via SCP
|
||||
- **Own conf in local cache**: copies own conf to `/tmp/varaverk/conf/${my_id}.conf`
|
||||
on full sync (so the cache has a complete snapshot of all confs)
|
||||
|
||||
Only partner confs are sourced from cache — `load_config.sh` always reads own conf
|
||||
@@ -136,7 +136,7 @@ A notification fires if any partner fails — check partner reachability via Tai
|
||||
|
||||
### What It Does
|
||||
|
||||
At array stop, copies all partner confs from `/tmp/.cache/vv/d/` to
|
||||
At array stop, copies all partner confs from `/tmp/varaverk/conf/` to
|
||||
`$PERSISTENT_CONF_CACHE`. Own conf is skipped (always on disk). The backup survives
|
||||
the reboot and is used by `conf_cache_restore.sh` at next array start if the sync
|
||||
can't reach the partner.
|
||||
@@ -164,7 +164,7 @@ conf_cache_save.sh --log # verbose output
|
||||
### What It Does
|
||||
|
||||
At array start (after `conf_sync.sh`), checks which partner confs are missing from
|
||||
`/tmp/.cache/vv/d/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
|
||||
`/tmp/varaverk/conf/`. For each missing conf, loads it from `$PERSISTENT_CONF_CACHE`
|
||||
if a backup exists there.
|
||||
|
||||
Always removes the persistent backup when done — whether used or not. On a normal
|
||||
@@ -434,8 +434,8 @@ REBOOT_VM_WAIT=30
|
||||
# STATE_DIR and PERSISTENT_CONF_CACHE are derived from SCRIPTS_DIR in master.conf.
|
||||
# They adapt to internal (/boot/config/plugins/varaverk) or appdata storage mode
|
||||
# (/mnt/user/appdata/Varaverk) automatically — do not hardcode paths.
|
||||
STATE_DIR="${SCRIPTS_DIR}/State_Files"
|
||||
PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
|
||||
STATE_DIR="${DATA_DIR}/state"
|
||||
PERSISTENT_CONF_CACHE="${CACHE_BACKUP_DIR}/conf"
|
||||
```
|
||||
|
||||
---
|
||||
@@ -446,7 +446,7 @@ PERSISTENT_CONF_CACHE="${SCRIPTS_DIR}/.cache/vv/d"
|
||||
|
||||
```bash
|
||||
# Check what's in the RAM cache:
|
||||
ls -la /tmp/.cache/vv/d/
|
||||
ls -la /tmp/varaverk/conf/
|
||||
|
||||
# Check what's in the persistent backup:
|
||||
ls -la "$PERSISTENT_CONF_CACHE/" # set SCRIPTS_DIR first or use full path
|
||||
|
||||
@@ -35,7 +35,7 @@ any container-connecting or continuous scripts start.
|
||||
**Partner Conf Lost Across Reboots When Partner is Down**
|
||||
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
|
||||
names, tier delays) to operate. The partner conf lives in a RAM cache at
|
||||
`/tmp/.cache/vv/d/` — wiped every reboot. At array start, `conf_sync.sh` pulls
|
||||
`/tmp/varaverk/conf/` — wiped every reboot. At array start, `conf_sync.sh` pulls
|
||||
a fresh copy from the partner. But if the partner is offline at boot time, the
|
||||
pull fails and fallback has no partner vars to work with.
|
||||
|
||||
@@ -120,16 +120,16 @@ Plugin/unraid/Watchdogs/System/
|
||||
```
|
||||
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
|
||||
│
|
||||
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/.cache/vv/d/
|
||||
│ push own conf to partner's /tmp/.cache/vv/d/
|
||||
├─ conf_sync.sh ← SSH/SCP: pull partner confs into /tmp/varaverk/conf/
|
||||
│ push own conf to partner's /tmp/varaverk/conf/
|
||||
├─ conf_cache_restore.sh ← if partner was down: load last-known-good conf from
|
||||
│ $PERSISTENT_CONF_CACHE into /tmp/.cache/vv/d/
|
||||
│ $PERSISTENT_CONF_CACHE into /tmp/varaverk/conf/
|
||||
├─ docker_syslog_filter.sh ← before any container starts (veth filter must be live)
|
||||
└─ inotify_tuning.sh ← before docker_network_connect.sh and continuous scripts
|
||||
|
||||
Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
|
||||
│
|
||||
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/.cache/vv/d/ → $PERSISTENT_CONF_CACHE
|
||||
├─ conf_cache_save.sh ← FIRST: snapshot /tmp/varaverk/conf/ → $PERSISTENT_CONF_CACHE
|
||||
│ while RAM cache is still fresh
|
||||
├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
|
||||
└─ ...other stop scripts...
|
||||
|
||||
@@ -521,7 +521,7 @@ at `$PERSISTENT_CONF_CACHE` while the partner is offline.
|
||||
**Remote online:** removes the persistent backup if one exists. It is not needed —
|
||||
`conf_sync.sh` will pull a fresh copy on the next boot. Silent when backup is already absent.
|
||||
|
||||
**Remote offline:** copies partner confs from the RAM cache (`/tmp/.cache/vv/d/`) to
|
||||
**Remote offline:** copies partner confs from the RAM cache (`$CONF_RAM_CACHE_DIR`, `/tmp/varaverk/conf/`) to
|
||||
`$PERSISTENT_CONF_CACHE`. Runs every 15 minutes, so the backup stays current throughout
|
||||
an extended outage. If this host reboots while the partner is still down,
|
||||
`conf_cache_restore.sh` will load the backup into RAM and fallback.sh will have
|
||||
@@ -821,8 +821,8 @@ storage_watchdog.sh --status
|
||||
### stability_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the reboot log (survives reboots — in State_Files/):
|
||||
cat /boot/config/plugins/varaverk/State_Files/system_watchdog_reboots.db
|
||||
# Check the reboot log (survives reboots — in data/state/):
|
||||
cat /boot/config/plugins/varaverk/data/state/system_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check syslog near the reboot time:
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# `data/` — everything Varaverk keeps on disk
|
||||
|
||||
This directory is the single on-disk root. If Varaverk persists something across a reboot, it is
|
||||
under here. Move `DATA_DIR` in `master.conf` and the whole tree follows.
|
||||
|
||||
This file is the only tracked thing in here. Everything else is runtime data, gitignored, and
|
||||
per-host — none of it syncs, none of it belongs in the repo, and losing any of it costs at most
|
||||
a rebuild.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
data/
|
||||
├── db/ statistics, histories, counters, blocklists — things that accumulate
|
||||
├── state/ runtime state for every script: watchdogs, fallback, transcode, setup
|
||||
├── ai/ retrieval index, operator memory, token ledger, filed bugs, saved chats
|
||||
├── cache/ persistent backups of the tmpfs caches — and only those
|
||||
│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand
|
||||
│ └── conf/ partner host*.conf snapshot (0700 — holds credentials)
|
||||
└── logs/ retained log output
|
||||
```
|
||||
|
||||
Each has a conf variable. Never hardcode a path into a script; use the variable, because the
|
||||
variable is what a storage-mode migration rewrites.
|
||||
|
||||
| Directory | Variable | Notes |
|
||||
|---|---|---|
|
||||
| `db/` | `DB_DIR` | |
|
||||
| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
|
||||
| `ai/` | `AI_DATA_DIR` | |
|
||||
| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
|
||||
| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
|
||||
|
||||
---
|
||||
|
||||
## What is NOT here
|
||||
|
||||
**The tmpfs caches.** They live under `VV_CACHE_ROOT` (`/tmp/varaverk/`) and must stay there.
|
||||
The WebGUI payload cache is read every one to five seconds by every open tab, and the arr item
|
||||
caches are rewritten by the hundred megabytes. On flash that is pointless write load for data
|
||||
whose entire purpose is to be cheap and disposable.
|
||||
|
||||
```
|
||||
/tmp/varaverk/
|
||||
├── api/ WebGUI payloads — monitor, arrs, ai
|
||||
├── arr/ arr payloads (backed up to data/cache/arr/)
|
||||
├── conf/ partner confs (backed up to data/cache/conf/)
|
||||
├── ai/ partner token ledgers — deliberately never backed up
|
||||
└── jobs/ in-flight AI answers and container actions (0700)
|
||||
```
|
||||
|
||||
**Locks.** `/tmp/unraid_locks/` — deliberately outside both roots, and deliberately on tmpfs so a
|
||||
lock cannot outlive the boot that took it.
|
||||
|
||||
**Configuration.** `Configurations/` holds the confs. Data and config stay apart: one is written
|
||||
by scripts, the other by you.
|
||||
|
||||
---
|
||||
|
||||
## Where a new file goes
|
||||
|
||||
Ask what happens if it is deleted.
|
||||
|
||||
- *Something is permanently lost* → it is a source of truth. `db/` if it accumulates, `state/`
|
||||
if it describes right now.
|
||||
- *It gets re-fetched and nothing else changes* → it is a cache. If it also lives in tmpfs, its
|
||||
backup goes in `cache/`. If it only lives here, it is not really a cache — put it in `db/`.
|
||||
- *Nothing at all* → it should not be written to disk in the first place.
|
||||
|
||||
`cache/` is the one that gets misused. It means "persistent backup of a tmpfs cache", not
|
||||
"anything cache-shaped". A file that is only ever here is a source of truth no matter what it is
|
||||
called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
|
||||
reason.
|
||||
|
||||
---
|
||||
|
||||
## History
|
||||
|
||||
Until 2026-08-08 this was two roots and two strays: `data/` and `State_Files/` as siblings, the
|
||||
conf-cache backup off in `SCRIPTS_DIR/.cache/vv/d`, and the arr backups loose in `data/`'s root.
|
||||
No single decision there was wrong. Together they meant nothing answered "what does Varaverk keep
|
||||
on disk", and the PHP layer — which cannot source bash — restated the paths it needed, so the two
|
||||
layers agreed only by hand.
|
||||
|
||||
State is data. It is the data that happens to describe right now, so it belongs under the same
|
||||
root as the rest.
|
||||
|
||||
`STATE_DIR` kept its name and changed only its value, which is why that restructure did not touch
|
||||
the 15 conf entries, 18 shell paths and 23 PHP paths built on it.
|
||||
|
||||
**Migrating an installation:** `Deployment/migrate_data_layout.sh --dry-run` first, then without
|
||||
the flag. It is idempotent, it moves rather than copies, and it rewrites the conf values that
|
||||
`conf_upgrade` deliberately will not touch. Each host runs it itself — this directory is
|
||||
gitignored, so the restructure travels as code and conf while the files stay put.
|
||||
Reference in New Issue
Block a user