Gmer4Lfe
96b907acbc
The card described the machine and said nothing about the thing whose dashboard it is. Cache size matters most: VV_CACHE_ROOT is under /tmp, which on Unraid is RAM, so it is shown against the rootfs percentage it counts against. Jobs are locks with a live process — four lock files were sitting there and one job was actually running.
145 lines
7.7 KiB
PHP
145 lines
7.7 KiB
PHP
<?php
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
// PURPOSE
|
|
// Full monitor payload. Every metric the monitor page renders — system, CPU, memory,
|
|
// network, GPUs, disks and pools, array, parity, UPS, containers, VMs, transcodes, watchdog
|
|
// and rsync summaries, and remote node roll-ups — in one document.
|
|
//
|
|
// OPERATIONAL MODEL
|
|
// Deliberately one large response rather than many small ones. Most of these metrics share
|
|
// an underlying source — the Unraid API, /proc, the same emhttp ini files — so collecting
|
|
// them together means one pass over each source instead of one per endpoint. The page
|
|
// renders from a single consistent snapshot; twenty parallel fetches would render from
|
|
// twenty slightly different moments.
|
|
//
|
|
// Served from a 300s cache written by Tools/api_cache_writer.sh, which runs every minute.
|
|
// The page therefore almost never pays for collection — the background writer does. ?live
|
|
// forces a fresh build for the refresh button.
|
|
//
|
|
// DESIGN PRINCIPLES
|
|
// The cache check happens before the heavy includes.
|
|
// Only include/config.php is loaded to reach vv_cache_read(). monitor.php, vms.php and
|
|
// docker_folders.php are required only after a miss, so a cache hit costs one file read
|
|
// rather than parsing three libraries.
|
|
//
|
|
// The API cache is pre-warmed once, on purpose.
|
|
// vv_api_data() is called before the payload is assembled so the API-first functions
|
|
// below it share a single GraphQL round trip instead of each making their own.
|
|
//
|
|
// Every field is a named function call, in render order.
|
|
// The payload is a flat map of key to collector. Adding a metric is adding a line, and
|
|
// nothing in the assembly depends on anything else in it — so one expensive or broken
|
|
// collector can be moved or removed without touching the others.
|
|
//
|
|
// Reports the API's own health alongside the data.
|
|
// _api_status travels with the payload, so the page can show that a section degraded to
|
|
// its local fallback rather than silently presenting less detail.
|
|
//
|
|
// OPERATIONAL SAFEGUARDS
|
|
// Read-only. Every function here observes; none of them start, stop, or change anything.
|
|
//
|
|
// Cache miss is distinguished from empty payload.
|
|
// vv_cache_read() returns null on a miss, expiry, or unparseable file, and the check is
|
|
// an explicit !== null — so a legitimately sparse payload is served from cache instead
|
|
// of being mistaken for a miss and forced onto the expensive path on every poll.
|
|
//
|
|
// Every collector degrades to empty rather than fatal.
|
|
// The library suppresses its filesystem reads and redirects stderr on every shell call,
|
|
// so absent hardware — no GPU, no UPS, no ZFS, no VMs — yields an empty section and an
|
|
// unrendered card. On a payload this wide that property is what keeps one missing
|
|
// subsystem from blanking the entire page.
|
|
//
|
|
// Remote collection degrades per node, so one dark partner costs its own card and nothing
|
|
// else.
|
|
//
|
|
// Known cost: a cache miss on a host with an unreachable partner pays the remote SSH
|
|
// timeouts inline. That is why the background writer exists and why the cache window is
|
|
// long — the miss path is the exception, not the design.
|
|
//
|
|
// REQUEST
|
|
// GET served from the 300s cache when one is present
|
|
// GET ?live bypass the cache and collect everything fresh
|
|
//
|
|
// RESPONSE
|
|
// A flat object of the keys listed in the assembly below, plus _api_status and ts.
|
|
//
|
|
// DEPENDS ON
|
|
// include/ai.php AI residency and index figures, only when vv_ai_ui_on()
|
|
// include/config.php vv_cache_read()
|
|
// include/common.php raw hardware metrics — system, cpu, mem, net, gpu, disks,
|
|
// ups, parity, containers, transcodes, remote roll-ups
|
|
// (loaded transitively through include/monitor.php)
|
|
// include/monitor.php the "is anything wrong" roll-ups — vv_partner_state(),
|
|
// vv_fallback_state(), vv_watchdog_summary(),
|
|
// vv_scripts_status(), vv_rsync_status()
|
|
// include/unraid_api.php vv_api_data(), vv_api_get_status()
|
|
// include/vms.php vv_get_vms()
|
|
// include/docker_folders.php vv_get_docker_folders()
|
|
// Tools/api_cache_writer.sh writes the cache this endpoint normally serves
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
header('Content-Type: application/json');
|
|
require_once dirname(__DIR__) . '/include/config.php';
|
|
|
|
if (!isset($_GET['live'])) {
|
|
$_vv_cached = vv_cache_read('monitor', 300);
|
|
if ($_vv_cached !== null) { echo json_encode($_vv_cached); exit; }
|
|
unset($_vv_cached);
|
|
}
|
|
|
|
require_once dirname(__DIR__) . '/include/monitor.php';
|
|
require_once dirname(__DIR__) . '/include/vms.php';
|
|
require_once dirname(__DIR__) . '/include/docker_folders.php';
|
|
|
|
// Pre-warm the API cache with one request (shared by all API-first functions below).
|
|
vv_api_data();
|
|
|
|
// AI residency, for the Monitor tab's AI row. Collected here rather than by that card polling
|
|
// api/ai.php on its own cycle: vv_ai_runtime_stats() calls out to Ollama and shells nvidia-smi,
|
|
// and this payload is written once a minute by the background writer — a card polling it
|
|
// directly would pay both costs every five seconds on every open tab.
|
|
//
|
|
// Null on any host that is not the AI host or has AI_ENABLED false, which is also what makes the
|
|
// row absent rather than empty there. Same shape as every other optional subsystem on this page.
|
|
// Reads the shared 'ai' cache the writer maintains and only collects on a miss. Landing here at
|
|
// all already means the monitor cache missed; paying a second full AI collection on top of that
|
|
// would make the slowest request on this page slower still, for figures a background writer
|
|
// refreshed under a minute ago.
|
|
$_vv_ai = null;
|
|
if (vv_ai_ui_on()) {
|
|
require_once dirname(__DIR__) . '/include/ai.php';
|
|
$_vv_ai = vv_ai_monitor_block(vv_ai_stats_cached(isset($_GET['live'])));
|
|
}
|
|
|
|
echo json_encode([
|
|
'system' => vv_system_info(),
|
|
'varaverk' => vv_varaverk_state(),
|
|
'fallback' => vv_fallback_state(),
|
|
'fallback_active' => vv_fallback_active(),
|
|
'partner' => vv_partner_state(),
|
|
'resources' => vv_system_resources(),
|
|
'cpu' => vv_cpu_per_core(),
|
|
'mem' => vv_memory_breakdown(),
|
|
'net' => vv_network_stats(),
|
|
'gpu' => vv_gpu_stats(),
|
|
'gpus' => vv_gpu_stats_all(),
|
|
'gpu_procs' => vv_gpu_processes(),
|
|
'containers' => vv_docker_containers(),
|
|
'stopped' => vv_docker_stopped(),
|
|
'transcode' => vv_transcode_sessions(),
|
|
'ups' => vv_ups_stats(),
|
|
'parity' => vv_parity_status(),
|
|
'storage' => vv_storage_pools(),
|
|
'array_disks' => vv_array_disks(),
|
|
'disk_io' => vv_disk_io_rates(),
|
|
'watchdog' => vv_watchdog_summary(),
|
|
'scripts' => vv_scripts_status(),
|
|
'rsync' => vv_rsync_status(),
|
|
'thresholds' => vv_disk_thresholds(),
|
|
'vms' => vv_get_vms(),
|
|
'docker_folders' => vv_get_docker_folders(),
|
|
'remote_hosts' => vv_remote_hosts_stats(),
|
|
'ai' => $_vv_ai,
|
|
'_api_status' => vv_api_get_status(),
|
|
'ts' => time(),
|
|
]);
|