Document today's restructures, and give data/ a README that ships with it
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,96 @@
|
||||
# `data/` — everything Varaverk keeps on disk
|
||||
|
||||
This directory is the single on-disk root. If Varaverk persists something across a reboot, it is
|
||||
under here. Move `DATA_DIR` in `master.conf` and the whole tree follows.
|
||||
|
||||
This file is the only tracked thing in here. Everything else is runtime data, gitignored, and
|
||||
per-host — none of it syncs, none of it belongs in the repo, and losing any of it costs at most
|
||||
a rebuild.
|
||||
|
||||
---
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
data/
|
||||
├── db/ statistics, histories, counters, blocklists — things that accumulate
|
||||
├── state/ runtime state for every script: watchdogs, fallback, transcode, setup
|
||||
├── ai/ retrieval index, operator memory, token ledger, filed bugs, saved chats
|
||||
├── cache/ persistent backups of the tmpfs caches — and only those
|
||||
│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand
|
||||
│ └── conf/ partner host*.conf snapshot (0700 — holds credentials)
|
||||
└── logs/ retained log output
|
||||
```
|
||||
|
||||
Each has a conf variable. Never hardcode a path into a script; use the variable, because the
|
||||
variable is what a storage-mode migration rewrites.
|
||||
|
||||
| Directory | Variable | Notes |
|
||||
|---|---|---|
|
||||
| `db/` | `DB_DIR` | |
|
||||
| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
|
||||
| `ai/` | `AI_DATA_DIR` | |
|
||||
| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
|
||||
| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
|
||||
|
||||
---
|
||||
|
||||
## What is NOT here
|
||||
|
||||
**The tmpfs caches.** They live under `VV_CACHE_ROOT` (`/tmp/varaverk/`) and must stay there.
|
||||
The WebGUI payload cache is read every one to five seconds by every open tab, and the arr item
|
||||
caches are rewritten by the hundred megabytes. On flash that is pointless write load for data
|
||||
whose entire purpose is to be cheap and disposable.
|
||||
|
||||
```
|
||||
/tmp/varaverk/
|
||||
├── api/ WebGUI payloads — monitor, arrs, ai
|
||||
├── arr/ arr payloads (backed up to data/cache/arr/)
|
||||
├── conf/ partner confs (backed up to data/cache/conf/)
|
||||
├── ai/ partner token ledgers — deliberately never backed up
|
||||
└── jobs/ in-flight AI answers and container actions (0700)
|
||||
```
|
||||
|
||||
**Locks.** `/tmp/unraid_locks/` — deliberately outside both roots, and deliberately on tmpfs so a
|
||||
lock cannot outlive the boot that took it.
|
||||
|
||||
**Configuration.** `Configurations/` holds the confs. Data and config stay apart: one is written
|
||||
by scripts, the other by you.
|
||||
|
||||
---
|
||||
|
||||
## Where a new file goes
|
||||
|
||||
Ask what happens if it is deleted.
|
||||
|
||||
- *Something is permanently lost* → it is a source of truth. `db/` if it accumulates, `state/`
|
||||
if it describes right now.
|
||||
- *It gets re-fetched and nothing else changes* → it is a cache. If it also lives in tmpfs, its
|
||||
backup goes in `cache/`. If it only lives here, it is not really a cache — put it in `db/`.
|
||||
- *Nothing at all* → it should not be written to disk in the first place.
|
||||
|
||||
`cache/` is the one that gets misused. It means "persistent backup of a tmpfs cache", not
|
||||
"anything cache-shaped". A file that is only ever here is a source of truth no matter what it is
|
||||
called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
|
||||
reason.
|
||||
|
||||
---
|
||||
|
||||
## History
|
||||
|
||||
Until 2026-08-08 this was two roots and two strays: `data/` and `State_Files/` as siblings, the
|
||||
conf-cache backup off in `SCRIPTS_DIR/.cache/vv/d`, and the arr backups loose in `data/`'s root.
|
||||
No single decision there was wrong. Together they meant nothing answered "what does Varaverk keep
|
||||
on disk", and the PHP layer — which cannot source bash — restated the paths it needed, so the two
|
||||
layers agreed only by hand.
|
||||
|
||||
State is data. It is the data that happens to describe right now, so it belongs under the same
|
||||
root as the rest.
|
||||
|
||||
`STATE_DIR` kept its name and changed only its value, which is why that restructure did not touch
|
||||
the 15 conf entries, 18 shell paths and 23 PHP paths built on it.
|
||||
|
||||
**Migrating an installation:** `Deployment/migrate_data_layout.sh --dry-run` first, then without
|
||||
the flag. It is idempotent, it moves rather than copies, and it rewrites the conf values that
|
||||
`conf_upgrade` deliberately will not touch. Each host runs it itself — this directory is
|
||||
gitignored, so the restructure travels as code and conf while the files stay put.
|
||||
Reference in New Issue
Block a user