163 lines
8.5 KiB
Markdown
163 lines
8.5 KiB
Markdown
# Monitor
|
||
|
||
Everything this host is doing right now, plus each partner. Nothing on this tab changes the
|
||
system except the container controls — the rest is observation.
|
||
|
||
Two poll rates, deliberately split: CPU, memory and network refresh every second; everything
|
||
else every five. The slower payload is served from a cache the background writer refreshes once
|
||
a minute, so polling faster could not make it newer — it only decides how soon the page notices.
|
||
|
||
**Cards you do not have do not appear.** No GPU, no UPS, no VMs, no partner — the card is absent
|
||
rather than showing zeros. An empty card would be a permanent reminder of nothing.
|
||
|
||
**The board re-flows on a narrow screen.** On a tablet or a phone the cards regroup into fewer,
|
||
wider columns rather than shrinking in place. Nothing is hidden and nothing is dropped — the same
|
||
cards are there in the same order, cut into different rows.
|
||
|
||
Below the full-width layout the page scrolls as a whole instead of each card scrolling inside
|
||
itself. A board that is taller than the screen has to scroll somewhere, and scrolling the page
|
||
once is better than being handed a screen of cards that are each too short to read.
|
||
|
||
**Cards in the same row share a height.** They line up along the bottom, so a card with little to
|
||
say carries empty space under it. That space means its neighbour is the taller card — not that a
|
||
reading is missing.
|
||
|
||
The System card is the one exception. On a narrow screen it stops after Version and the Varaverk
|
||
figures below scroll, so that the host itself — hostname, clock, array state, uptime — is what
|
||
you get without scrolling anything.
|
||
|
||
---
|
||
|
||
## Reference — System, power, CPU, memory, network
|
||
|
||
| Card | What it tells you |
|
||
|---|---|
|
||
| System | Hostname, Unraid version, uptime, array state |
|
||
| Power | UPS load, battery and runtime. Absent when no UPS is configured |
|
||
| CPU | Per-core utilisation with a 2-minute history, plus package temperature |
|
||
| Memory | Used / cached / free, broken out rather than a single percentage |
|
||
| Network | Live rx/tx with a 2-minute history |
|
||
|
||
## Reference — Jobs, fallback, partner, containers
|
||
|
||
| Card | What it tells you |
|
||
|---|---|
|
||
| Scripts | What ran recently and how it exited. Click a row to open its log |
|
||
| Fallback | `NORMAL`, `FALLBACK`, `NO_INTERNET` or `DARK` — which tier this host is in |
|
||
| Partner | Each partner's uptime, load, memory and array state, over the mesh |
|
||
| Containers & VMs | Grouped by folder. Start and stop from here; stopping asks first |
|
||
|
||
## Reference — Sync, GPUs, transcodes, streams
|
||
|
||
| Card | What it tells you |
|
||
|---|---|
|
||
| Rsync | Last run per share, duration and outcome |
|
||
| GPU | Utilisation, VRAM and temperature, one card per GPU |
|
||
| Transcode | Ramdisk usage, SSD fallback, and the sessions currently running |
|
||
| Streams | What is playing now across Emby, Jellyfin and Plex |
|
||
|
||
## Reference — Health and storage
|
||
|
||
| Card | What it tells you |
|
||
|---|---|
|
||
| Watchdog | Roll-up across every watchdog. Healthy requires *every* strike set to be clear |
|
||
| Parity | Last check, its result, and when the next one is due |
|
||
| Pools | Each cache pool with its per-disk breakdown and live I/O |
|
||
| Array | Every array disk, its temperature against the warning thresholds, and live I/O |
|
||
|
||
## Reference — AI
|
||
|
||
Present only on the AI host with `$AI_ENABLED` true. Absent everywhere else, on the same rule
|
||
as the GPU and UPS cards.
|
||
|
||
| Card | What it tells you |
|
||
|---|---|
|
||
| AI | Model residency, context size, VRAM, index size and staleness |
|
||
| Tokens | What the assistant has spent — per node, and for today, the last 7 days and all time |
|
||
| Conversations | The last `$AI_CHAT_HISTORY_MAX` chats. Click to reopen, × to delete |
|
||
| Assistant | Ask a question without leaving the dashboard |
|
||
| Profile chip | Which contract answers — General Chat, Varaverk Assistant, Code Sketcher, Troubleshoot |
|
||
| ⤢ | Expands the conversation, and collapses it again. Always opens collapsed |
|
||
| Medium / Large | In the banner, once expanded — how much room expanding is worth |
|
||
| Shortcuts | In the banner — the keyboard list, paid for out of the transcript rather than added to it |
|
||
| Saved | Saved conversations, the same ones the Conversations card lists |
|
||
|
||
---
|
||
|
||
## The AI card leads with offload, not with size
|
||
|
||
`GPU offload` is the number worth watching. At 100% the model is fully resident and answers at
|
||
roughly 60 tokens per second on this hardware; with even a few layers pushed to CPU that falls
|
||
to about a quarter of it. Nothing else in the WebGUI surfaces it, and a model that has quietly
|
||
fallen back to partial offload is invisible until answers merely *feel* slow.
|
||
|
||
`Index` reports chunks and whether the index is stale. Stale means a tracked file is newer than
|
||
the last build — the assistant will still answer confidently, out of code that has since changed.
|
||
That is the one failure a grounded answer cannot reveal on its own, so it is stated rather than
|
||
implied.
|
||
|
||
These figures come from the same one-minute collection the AI tab's banner uses, so the two
|
||
cannot disagree about whether the model is loaded.
|
||
|
||
## Tokens is its own card, not a line on the AI one
|
||
|
||
The AI card answers whether the model is healthy right now. What it has spent over a month is a
|
||
different question, and it was crowding that card out one line at a time.
|
||
|
||
`Nodes` lists each host with its total. A host with no rows reads **not collected here**, never
|
||
zero — each host writes to its own `data/` and only `ai_token_sync.sh` moves a ledger between
|
||
them, so a zero would claim the partner sat idle when the truth is that this host cannot see its
|
||
ledger at all. Click a node to scope the figures below it to that host; click **All hosts** to go
|
||
back.
|
||
|
||
The totals cover today, the last 7 days and all time, each with the turn count and the prompt /
|
||
completion split. The footer adds how long the ledger has been running, the best throughput seen,
|
||
and — under All hosts only — the split by profile and by source. Those last two describe the whole
|
||
ledger, so they are not shown under a single host's heading where they would not describe it.
|
||
|
||
This is the same ledger the AI tab shows, read from the same endpoint. It refreshes when a turn
|
||
finishes here, and otherwise once a minute, because the numbers only move when a turn completes.
|
||
|
||
## The assistant here starts on General Chat
|
||
|
||
The AI tab starts on Varaverk Assistant, which answers only from this installation's
|
||
documentation and cites its sources. This card starts on General Chat instead, because the jobs
|
||
are different: the tab is where you go to interrogate the system, this is the box you type an
|
||
idle question into while watching the dashboard.
|
||
|
||
Nothing is lost by starting loose. A question that is genuinely about this installation is handed
|
||
to the strict profile automatically, and handed back if the documentation turns out not to cover
|
||
it. Starting strict would refuse ordinary questions to guard against a mistake the server already
|
||
prevents.
|
||
|
||
Switching profile with the chip keeps what is on screen but stops sending earlier turns to the
|
||
model. Carrying cited, retrieval-grounded answers into a mode with no retrieval makes the model
|
||
keep referring to sources it can no longer see.
|
||
|
||
**Troubleshoot from here has no log open.** On the Scheduler tab it reads whatever log you have
|
||
in front of you; picked from the dashboard there is nothing attached, so it says so and tells you
|
||
how to give it evidence — open the log on the Scheduler tab, or name the script in the question,
|
||
which resolves that script's run record and log on its own. What it will not do is describe a log
|
||
it was never given.
|
||
|
||
## Conversations are shared with the AI tab
|
||
|
||
The same store backs both. Start a thread here, finish it on the AI tab, or the reverse — it is
|
||
one conversation, not two. Chats save themselves when a turn completes; there is no Save button,
|
||
because a conversation worth keeping is not reliably one you knew was worth keeping while you
|
||
were having it. The oldest drops off past `$AI_CHAT_HISTORY_MAX`.
|
||
|
||
Reopening a chat shows the turns as plain text. Sources, reasoning and timings describe one
|
||
generation and are not stored — redrawing them beside a transcript you may continue under a
|
||
different profile would be citing evidence for an answer that is no longer being made.
|
||
|
||
## What this tab will not do
|
||
|
||
It will not clear a watchdog strike, reset a counter, or start a parity check. Strike state
|
||
belongs to the watchdog that owns it, and a page that cleared strikes would silently undo an
|
||
escalation something was deliberately building toward.
|
||
|
||
Stopping a container asks first. Starting one does not — and either way the endpoint validates
|
||
the request against real inventory, so a stale page cannot act on a container that no longer
|
||
exists.
|