Varaverk/data/README.md

data/ — everything Varaverk keeps on disk

This directory is the single on-disk root. If Varaverk persists something across a reboot, it is under here. Move DATA_DIR in master.conf and the whole tree follows.

This file is the only tracked thing in here. Everything else is runtime data, gitignored, and per-host — none of it syncs, none of it belongs in the repo, and losing any of it costs at most a rebuild.

---

Layout

data/
├── db/       statistics, histories, counters, blocklists — things that accumulate
├── state/    runtime state for every script: watchdogs, fallback, transcode, setup
├── ai/       retrieval index, operator memory, token ledger, filed bugs, saved chats
├── cache/    persistent backups of the tmpfs caches — and only those
│   ├── arr/    *_tracked_cache.json, restored into tmpfs on demand
│   └── conf/   partner host*.conf snapshot (0700 — holds credentials)
├── Backups/  point-in-time copies kept so a bad write can be undone
│   └── Confs/  pre-write copies of this host's own confs (0700 — holds credentials)
└── logs/     retained log output

Each has a conf variable. Never hardcode a path into a script; use the variable, because the variable is what a storage-mode migration rewrites.

Directory	Variable	Notes
db/	DB_DIR
state/	STATE_DIR	ALL state files must use this. No /tmp, no repo root.
ai/	AI_DATA_DIR
cache/	CACHE_BACKUP_DIR	ARR_CACHE_BACKUP_DIR, PERSISTENT_CONF_CACHE sit under it
Backups/	BACKUP_DIR	parent only; each kind of backup gets a subdirectory
Backups/Confs/	CONF_BACKUP_DIR	retained per CONF_BACKUP_RETAIN; not a cache — see below
logs/	LOG_ARCHIVE_DIR	live logging still goes to LOG_DIR (/var/log/varaverk)

---

What is NOT here

The tmpfs caches. They live under VV_CACHE_ROOT (/tmp/varaverk/) and must stay there. The WebGUI payload cache is read every one to five seconds by every open tab, and the arr item caches are rewritten by the hundred megabytes. On flash that is pointless write load for data whose entire purpose is to be cheap and disposable.

/tmp/varaverk/
├── api/    WebGUI payloads — monitor, arrs, ai
├── arr/    arr payloads (backed up to data/cache/arr/)
├── conf/   partner confs   (backed up to data/cache/conf/)
├── ai/     partner token ledgers — deliberately never backed up
└── jobs/   in-flight AI answers and container actions (0700)

Locks. /tmp/unraid_locks/ — deliberately outside both roots, and deliberately on tmpfs so a lock cannot outlive the boot that took it.

Configuration. Configurations/ holds the confs. Data and config stay apart: one is written by scripts, the other by you.

---

Where a new file goes

Ask what happens if it is deleted.

• Something is permanently lost → it is a source of truth. db/ if it accumulates, state/ if it describes right now.
• It gets re-fetched and nothing else changes → it is a cache. If it also lives in tmpfs, its backup goes in cache/. If it only lives here, it is not really a cache — put it in db/.
• Nothing at all → it should not be written to disk in the first place.

cache/ is the one that gets misused. It means "persistent backup of a tmpfs cache", not "anything cache-shaped". A file that is only ever here is a source of truth no matter what it is called — lidarr_art_miss_cache.tsv has "cache" in its name and lives in db/ for exactly that reason.

Backups/ is the same trap from the other side: it has "backup" in its name but backs up nothing that exists elsewhere. The confs are gitignored, so a pre-write copy under Backups/Confs/ is the only prior version of that file anywhere. Deleting it loses something permanently, which is why it is a root of its own and not a subdirectory of cache/.

---

History

Until 2026-08-08 this was two roots and two strays: data/ and State_Files/ as siblings, the conf-cache backup off in SCRIPTS_DIR/.cache/vv/d, and the arr backups loose in data/'s root. No single decision there was wrong. Together they meant nothing answered "what does Varaverk keep on disk", and the PHP layer — which cannot source bash — restated the paths it needed, so the two layers agreed only by hand.

State is data. It is the data that happens to describe right now, so it belongs under the same root as the rest.

STATE_DIR kept its name and changed only its value, which is why that restructure did not touch the 15 conf entries, 18 shell paths and 23 PHP paths built on it.

Migrating an installation: Deployment/migrate_data_layout.sh --dry-run first, then without the flag. It is idempotent, it moves rather than copies, and it rewrites the conf values that conf_upgrade deliberately will not touch. Each host runs it itself — this directory is gitignored, so the restructure travels as code and conf while the files stay put.