Gmer4Lfe
d9f917ecef
These files are gitignored, so a bad write had nothing to go back to; the escaping order also stored any value containing a quote truncated but still parseable, which only a read-back can catch.
106 lines
4.9 KiB
Markdown
106 lines
4.9 KiB
Markdown
# `data/` — everything Varaverk keeps on disk
|
|
|
|
This directory is the single on-disk root. If Varaverk persists something across a reboot, it is
|
|
under here. Move `DATA_DIR` in `master.conf` and the whole tree follows.
|
|
|
|
This file is the only tracked thing in here. Everything else is runtime data, gitignored, and
|
|
per-host — none of it syncs, none of it belongs in the repo, and losing any of it costs at most
|
|
a rebuild.
|
|
|
|
---
|
|
|
|
## Layout
|
|
|
|
```
|
|
data/
|
|
├── db/ statistics, histories, counters, blocklists — things that accumulate
|
|
├── state/ runtime state for every script: watchdogs, fallback, transcode, setup
|
|
├── ai/ retrieval index, operator memory, token ledger, filed bugs, saved chats
|
|
├── cache/ persistent backups of the tmpfs caches — and only those
|
|
│ ├── arr/ *_tracked_cache.json, restored into tmpfs on demand
|
|
│ └── conf/ partner host*.conf snapshot (0700 — holds credentials)
|
|
├── Backups/ point-in-time copies kept so a bad write can be undone
|
|
│ └── Confs/ pre-write copies of this host's own confs (0700 — holds credentials)
|
|
└── logs/ retained log output
|
|
```
|
|
|
|
Each has a conf variable. Never hardcode a path into a script; use the variable, because the
|
|
variable is what a storage-mode migration rewrites.
|
|
|
|
| Directory | Variable | Notes |
|
|
|---|---|---|
|
|
| `db/` | `DB_DIR` | |
|
|
| `state/` | `STATE_DIR` | ALL state files must use this. No `/tmp`, no repo root. |
|
|
| `ai/` | `AI_DATA_DIR` | |
|
|
| `cache/` | `CACHE_BACKUP_DIR` | `ARR_CACHE_BACKUP_DIR`, `PERSISTENT_CONF_CACHE` sit under it |
|
|
| `Backups/` | `BACKUP_DIR` | parent only; each kind of backup gets a subdirectory |
|
|
| `Backups/Confs/` | `CONF_BACKUP_DIR` | retained per `CONF_BACKUP_RETAIN`; **not** a cache — see below |
|
|
| `logs/` | `LOG_ARCHIVE_DIR` | live logging still goes to `LOG_DIR` (`/var/log/varaverk`) |
|
|
|
|
---
|
|
|
|
## What is NOT here
|
|
|
|
**The tmpfs caches.** They live under `VV_CACHE_ROOT` (`/tmp/varaverk/`) and must stay there.
|
|
The WebGUI payload cache is read every one to five seconds by every open tab, and the arr item
|
|
caches are rewritten by the hundred megabytes. On flash that is pointless write load for data
|
|
whose entire purpose is to be cheap and disposable.
|
|
|
|
```
|
|
/tmp/varaverk/
|
|
├── api/ WebGUI payloads — monitor, arrs, ai
|
|
├── arr/ arr payloads (backed up to data/cache/arr/)
|
|
├── conf/ partner confs (backed up to data/cache/conf/)
|
|
├── ai/ partner token ledgers — deliberately never backed up
|
|
└── jobs/ in-flight AI answers and container actions (0700)
|
|
```
|
|
|
|
**Locks.** `/tmp/unraid_locks/` — deliberately outside both roots, and deliberately on tmpfs so a
|
|
lock cannot outlive the boot that took it.
|
|
|
|
**Configuration.** `Configurations/` holds the confs. Data and config stay apart: one is written
|
|
by scripts, the other by you.
|
|
|
|
---
|
|
|
|
## Where a new file goes
|
|
|
|
Ask what happens if it is deleted.
|
|
|
|
- *Something is permanently lost* → it is a source of truth. `db/` if it accumulates, `state/`
|
|
if it describes right now.
|
|
- *It gets re-fetched and nothing else changes* → it is a cache. If it also lives in tmpfs, its
|
|
backup goes in `cache/`. If it only lives here, it is not really a cache — put it in `db/`.
|
|
- *Nothing at all* → it should not be written to disk in the first place.
|
|
|
|
`cache/` is the one that gets misused. It means "persistent backup of a tmpfs cache", not
|
|
"anything cache-shaped". A file that is only ever here is a source of truth no matter what it is
|
|
called — `lidarr_art_miss_cache.tsv` has "cache" in its name and lives in `db/` for exactly that
|
|
reason.
|
|
|
|
`Backups/` is the same trap from the other side: it has "backup" in its name but backs up nothing
|
|
that exists elsewhere. The confs are gitignored, so a pre-write copy under `Backups/Confs/` is the
|
|
only prior version of that file anywhere. Deleting it loses something permanently, which is why it
|
|
is a root of its own and not a subdirectory of `cache/`.
|
|
|
|
---
|
|
|
|
## History
|
|
|
|
Until 2026-08-08 this was two roots and two strays: `data/` and `State_Files/` as siblings, the
|
|
conf-cache backup off in `SCRIPTS_DIR/.cache/vv/d`, and the arr backups loose in `data/`'s root.
|
|
No single decision there was wrong. Together they meant nothing answered "what does Varaverk keep
|
|
on disk", and the PHP layer — which cannot source bash — restated the paths it needed, so the two
|
|
layers agreed only by hand.
|
|
|
|
State is data. It is the data that happens to describe right now, so it belongs under the same
|
|
root as the rest.
|
|
|
|
`STATE_DIR` kept its name and changed only its value, which is why that restructure did not touch
|
|
the 15 conf entries, 18 shell paths and 23 PHP paths built on it.
|
|
|
|
**Migrating an installation:** `Deployment/migrate_data_layout.sh --dry-run` first, then without
|
|
the flag. It is idempotent, it moves rather than copies, and it rewrites the conf values that
|
|
`conf_upgrade` deliberately will not touch. Each host runs it itself — this directory is
|
|
gitignored, so the restructure travels as code and conf while the files stay put.
|