FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Measure the link, not rsync: Data Transferred now reads Tailscale per-peer counters

Browse Source 
The old card totalled rsync's own logs, so it reported 'no data moved' across a link that had carried hundreds of gigabytes over SSH, the arr APIs, conf pushes and the Unraid API.
This commit is contained in:
Gmer4Lfe
2026-08-17 16:17:11 -04:00
parent 25e055a7ac
commit fbf6a07da9
4 changed files with 275 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Plugin/unraid/Tools/api_cache_writer.sh
+6
View File
@@ -68,3 +68,9 @@
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
php "$SCRIPT_DIR/api_cache_writer.php"
# Mesh traffic sample. Rides this job because it needs a steady once-a-minute cadence and adding
# a second per-minute cron for one append is more moving parts than the measurement is worth.
# Failure is ignored on purpose: a missed sample costs resolution in one window, and this job's
# actual purpose is the WebGUI cache.
php "$SCRIPT_DIR/mesh_traffic_sample.php" >/dev/null 2>&1 || true
Plugin/unraid/Tools/mesh_traffic_sample.php
+116
View File
@@ -0,0 +1,116 @@
<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Sample per-peer Tailscale byte counters, once a minute, so the Partnership page can report
// how much has actually moved between the servers in this mesh over a window.
//
// OPERATIONAL MODEL
// Appends one line per mesh peer to data/db/mesh_traffic.db:
//
// epoch|peer-hostname|txbytes|rxbytes
//
// Called from Tools/api_cache_writer.sh, which already runs every minute. A window total is
// then the difference between the newest sample and the oldest one still inside that window.
//
// DESIGN PRINCIPLES
// Measure the link, not the tool.
// The previous card totalled rsync's own logs, so it could only ever describe rsync — and
// said "no data moved" while SSH, the arr APIs, conf pushes and the Unraid API were all
// using the same link. Tailscale counts the bytes on the wire, whatever sent them.
//
// Mesh peers only.
// The tailnet holds phones and workstations. Filtered against the HOST* hostnames in
// master.conf so this measures the partnership, not the tailnet.
//
// Absolute counters are stored, never deltas.
// A delta computed at write time bakes in whatever the sampling interval happened to be
// and cannot be re-derived if a run is missed. Storing the raw counter means a gap costs
// resolution, not correctness.
//
// OPERATIONAL SAFEGUARDS
// Counter resets are the reader's problem, not this file's — tailscaled restarting returns
// the counters to zero, and a sampler that tried to compensate would have to guess when.
// Storing raw values leaves the evidence intact: a sample lower than the one before it is a
// restart, and it is unambiguous.
//
// Trimmed to VV_MESH_KEEP_DAYS on every run, so the file cannot grow without bound. At one
// sample per peer per minute that is ~1,440 lines/peer/day.
//
// Silent no-op when tailscale is absent or returns nothing parseable. This runs every minute
// from a cache writer; a mesh sampler is not worth a log line per minute when it has nothing
// to add.
//
// RUNTIME MODES
// php mesh_traffic_sample.php append one sample per mesh peer
// php mesh_traffic_sample.php --show print what it would record, write nothing
//
// DEPENDS ON
// tailscale status --json the counters
// include/config.php DATA_DIR, vv_conf_vars() for the HOST* list
// ═══════════════════════════════════════════════════════════════════════════════════════════════
require_once dirname(__DIR__) . '/include/config.php';
define('VV_MESH_DB', DATA_DIR . '/db/mesh_traffic.db');
define('VV_MESH_KEEP_DAYS', 40); // a 30-day window needs a sample older than 30 days
$show = in_array('--show', $argv ?? [], true);
$raw = shell_exec('tailscale status --json 2>/dev/null');
$js = json_decode((string)$raw, true);
if (!is_array($js) || empty($js['Peer'])) exit(0);
// The hostnames this mesh is made of. Anything else on the tailnet is somebody's laptop.
$vars = vv_conf_vars();
$mesh = [];
foreach ($vars as $k => $v) {
if (preg_match('/^HOST\d+$/', $k) && trim((string)$v) !== '') $mesh[strtolower(trim($v))] = true;
}
if (!$mesh) exit(0);
$now = time();
$lines = [];
foreach ($js['Peer'] as $peer) {
$host = strtolower(trim((string)($peer['HostName'] ?? '')));
if ($host === '') continue;
// Same unambiguous-prefix rule the rest of the partnership layer uses: the tailnet name and
// the conf hostname are different strings and nothing keeps them in step.
$match = null;
if (isset($mesh[$host])) {
$match = $host;
} else {
$cand = [];
foreach (array_keys($mesh) as $m) {
if (str_starts_with($host, $m) || str_starts_with($m, $host)) $cand[] = $m;
}
if (count($cand) === 1) $match = $cand[0];
}
if ($match === null) continue;
$tx = (int)($peer['TxBytes'] ?? 0);
$rx = (int)($peer['RxBytes'] ?? 0);
if ($tx === 0 && $rx === 0) continue;
$lines[] = $now . '|' . $match . '|' . $tx . '|' . $rx;
}
if (!$lines) exit(0);
if ($show) { echo implode("\n", $lines) . "\n"; exit(0); }
@mkdir(dirname(VV_MESH_DB), 0755, true);
@file_put_contents(VV_MESH_DB, implode("\n", $lines) . "\n", FILE_APPEND | LOCK_EX);
// Trim in place. Read-filter-rewrite rather than append-only truncation, because the cut is by
// age and the file is not ordered by peer.
$cutoff = $now - (VV_MESH_KEEP_DAYS * 86400);
$all = @file(VV_MESH_DB, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES) ?: [];
if (count($all) > 200) {
$keep = [];
foreach ($all as $l) {
$ts = (int)strtok($l, '|');
if ($ts >= $cutoff) $keep[] = $l;
}
if (count($keep) !== count($all)) {
$tmp = VV_MESH_DB . '.tmp';
if (@file_put_contents($tmp, implode("\n", $keep) . "\n") !== false) @rename($tmp, VV_MESH_DB);
}
}
Plugin/unraid/include/partnership.php
+74 -51
View File
@@ -240,68 +240,91 @@ function vv_pt_remote_system(string $ip, string $sshKey): array {
];
}
// ── Transfer volume ───────────────────────────────────────────────────────────
//
// Reads data/db/bandwidth_history.db, one line per completed rsync:
// YYYY-MM-DD|HH:MM|profile|duration|status|bytes
//
// Rows before VV_BW_EPOCH are excluded outright. Until 2026-08-14 rsync.sh read the byte count
// from the wrong field of rsync --stats and every row was written as 0 — 1,423 of them. Summing
// those produces a confident 0.00 GB for any window that reaches back far enough, which is worse
// than an empty card: it is a measurement that looks taken. The cut is on the fix, not on a
// guess about which rows look plausible.
//
// A window whose rows are all zero reports null, not 0, so the UI can say "no transfers
// recorded" rather than assert nothing moved. Those are different claims and only one of them
// is supported by an all-zero column.
define('VV_BW_EPOCH', '2026-08-14');
function vv_pt_transfer_stats(): array {
$file = DATA_DIR . '/db/bandwidth_history.db';
$out = ['epoch' => VV_BW_EPOCH, 'windows' => [], 'live' => null, 'usable_rows' => 0];
// ── Mesh traffic ──────────────────────────────────────────────────────────────
//
// What has actually crossed the link to each partner, from Tailscale's own per-peer byte
// counters — not from rsync's logs.
//
// The previous version totalled data/db/bandwidth_history.db, so it could only ever describe
// rsync. Everything else using the same link — SSH, the arr APIs, conf pushes, the Unraid API,
// the webhook — was invisible to it, and it reported "no data moved" across a link carrying
// hundreds of gigabytes. Counting bytes on the wire measures the partnership rather than one
// tool's opinion of itself.
//
// Windows come from Tools/mesh_traffic_sample.php, which records the raw counters once a
// minute. A window total is the newest sample minus the oldest one still inside it.
function vv_pt_mesh_traffic(): array {
$file = DATA_DIR . '/db/mesh_traffic.db';
$out = ['peers' => [], 'samples' => 0];
if (!is_file($file)) return $out;
$windows = ['24h' => 86400, '7d' => 604800, '30d' => 2592000];
foreach ($windows as $k => $_) $out['windows'][$k] = ['bytes' => null, 'runs' => 0, 'failed' => 0];
if (is_file($file)) {
$now = time();
$epoch = strtotime(VV_BW_EPOCH . ' 00:00:00');
$byPeer = [];
$fh = fopen($file, 'r');
if ($fh) {
if (!$fh) return $out;
while (($line = fgets($fh)) !== false) {
$p = explode('|', trim($line));
if (count($p) < 6) continue;
$ts = strtotime($p[0] . ' ' . $p[1]);
if (!$ts || $ts < $epoch) continue;
$out['usable_rows']++;
$bytes = (int)$p[5];
$failed = ($p[4] ?? '') !== 'success';
foreach ($windows as $k => $span) {
if ($ts < $now - $span) continue;
$w = &$out['windows'][$k];
$w['runs']++;
if ($failed) $w['failed']++;
if ($bytes > 0) $w['bytes'] = (int)$w['bytes'] + $bytes;
unset($w);
}
if (count($p) < 4) continue;
$byPeer[$p[1]][] = [(int)$p[0], (int)$p[2], (int)$p[3]];
$out['samples']++;
}
fclose($fh);
$now = time();
$windows = ['24h' => 86400, '7d' => 604800, '30d' => 2592000];
foreach ($byPeer as $peer => $rows) {
usort($rows, fn($a, $b) => $a[0] <=> $b[0]);
$last = end($rows);
// Live rate from the two most recent samples. Meaningless if they are far apart — a gap
// means the sampler missed runs, and dividing by that gap reports an average over a
// period nobody watched as though it were current.
$live = null;
$n = count($rows);
if ($n >= 2) {
$prev = $rows[$n - 2];
$dt = $last[0] - $prev[0];
if ($dt > 0 && $dt <= 300 && $last[1] >= $prev[1] && $last[2] >= $prev[2]) {
$live = ['tx_bps' => (int)(($last[1] - $prev[1]) / $dt),
'rx_bps' => (int)(($last[2] - $prev[2]) / $dt),
'age' => $now - $last[0]];
}
}
// Live: an rsync this host is running right now. The lock name carries the share, which is
// the only part of "what is moving" worth showing without parsing progress output.
$running = trim((string)shell_exec("pgrep -c -f '^rsync -av' 2>/dev/null")) ;
if ((int)$running > 0) {
$share = '';
foreach (glob('/tmp/unraid_locks/rsync_*.lock') ?: [] as $l) {
$pid = (int)preg_replace('/\D/', '', (string)@file_get_contents($l));
if ($pid && is_dir("/proc/$pid")) {
$share = preg_replace('/^rsync_|\.lock$/', '', basename($l));
break;
$win = [];
foreach ($windows as $k => $span) {
$from = $now - $span;
// Sum forward through the window rather than subtracting endpoints, so a tailscaled
// restart — which returns the counters to zero — costs one interval instead of
// producing a negative total or a spike the size of the whole previous session.
$tx = 0; $rx = 0; $seen = 0; $oldest = null;
$prev = null;
foreach ($rows as $r) {
if ($r[0] < $from) { $prev = $r; continue; }
if ($oldest === null) $oldest = $r[0];
if ($prev !== null) {
$tx += ($r[1] >= $prev[1]) ? $r[1] - $prev[1] : $r[1];
$rx += ($r[2] >= $prev[2]) ? $r[2] - $prev[2] : $r[2];
$seen++;
}
$prev = $r;
}
$out['live'] = ['running' => (int)$running, 'share' => $share];
$win[$k] = [
'tx' => $tx, 'rx' => $rx, 'intervals' => $seen,
// How far back the data actually reaches. A 30-day figure built from six hours of
// samples is not a 30-day figure, and the card has to be able to say so.
'covers' => $oldest ? $now - $oldest : 0,
];
}
$out['peers'][$peer] = [
'live' => $live,
'windows' => $win,
// Counter totals since tailscaled last started — the longest view available without
// any history at all, and the one that is right on a fresh install.
'session' => ['tx' => $last[1], 'rx' => $last[2]],
];
}
return $out;
}
@@ -624,7 +647,7 @@ function vv_partnership_all(): array {
'config' => vv_pt_config(),
'nodes' => vv_pt_nodes(),
'sync' => vv_pt_sync(),
'xfer' => vv_pt_transfer_stats(),
'xfer' => vv_pt_mesh_traffic(),
'ts' => time(),
];
}
Plugin/unraid/pages/partnership.php
+72 -50
View File
@@ -598,62 +598,84 @@ function _renderOfflineWarn(cfg) {
// ── Data transferred ──────────────────────────────────────────────────────────
//
// A window with runs but no bytes says "no data moved", not "0.00 GB". They are different
// claims: rsync completing with nothing to send is the normal state for a mirror that is
// already in step, and printing a zero total invites the reading that the sync is broken.
// A window with no runs at all says so separately again.
function _renderXfer(x) {
const el = document.getElementById('vv-pt-xfer-body');
if (!el) return;
x = x || {};
const w = x.windows || {};
const fmt = (b) => {
if (b === null || b === undefined) return null;
// Tailscale's per-peer counters, so this is every byte across the link — rsync, SSH, the arr
// APIs, conf pushes, the Unraid API — not one tool's log of itself.
//
// A window says how far its samples actually reach. The sampler starts collecting the first
// minute this ships, so "30 days" means thirty days of history only after thirty days of it;
// until then the figure is real but covers less than the label, and saying which is the
// difference between a number and a claim.
function _vvBytes(b) {
if (!b) return '0 B';
const u = ['B','KB','MB','GB','TB'];
let i = 0, v = b;
while (v >= 1024 && i < u.length - 1) { v /= 1024; i++; }
return (i >= 3 ? v.toFixed(2) : Math.round(v)) + ' ' + u[i];
};
const row = (label, key) => {
const d = w[key] || {};
const bytes = fmt(d.bytes);
let val, col;
if (!d.runs) { val = 'no syncs ran'; col = '#333'; }
else if (bytes) { val = bytes; col = '#4caf50'; }
else { val = 'no data moved'; col = '#555'; }
const extra = d.runs
? `<span style="color:#333;font-size:9px;">${d.runs} run${d.runs===1?'':'s'}${d.failed?` · ${d.failed} failed`:''}</span>`
: '';
return `<div style="display:flex;justify-content:space-between;align-items:baseline;gap:10px;
padding:5px 0;border-bottom:1px solid #171717;">
<span style="color:#666;font-size:11px;">${label}</span>
<span style="display:flex;align-items:baseline;gap:8px;">
${extra}<span style="color:${col};font-size:12px;font-family:monospace;">${val}</span>
</span></div>`;
};
const live = x.live;
const liveHtml = live
? `<div style="display:flex;justify-content:space-between;align-items:baseline;gap:10px;
padding:5px 0;border-bottom:1px solid #171717;">
<span style="color:#666;font-size:11px;">Live</span>
<span style="color:#ff9800;font-size:12px;font-family:monospace;">⟳ ${live.running} transfer${live.running===1?'':'s'}${live.share?` · ${vvEscHtml(live.share)}`:''}</span>
</div>`
: `<div style="display:flex;justify-content:space-between;align-items:baseline;gap:10px;
padding:5px 0;border-bottom:1px solid #171717;">
<span style="color:#666;font-size:11px;">Live</span>
<span style="color:#333;font-size:12px;font-family:monospace;">idle</span>
</div>`;
el.innerHTML = liveHtml + row('Last 24 hours','24h') + row('Last 7 days','7d') + row('Last 30 days','30d')
+ `<div style="margin-top:7px;font-size:9px;color:#2e2e2e;line-height:1.5;">
Counted from ${vvEscHtml(x.epoch || '')} — before that rsync logged every transfer as
0 bytes, so those rows are excluded rather than summed into a false total.
</div>`;
}
function _vvDur(s) {
if (!s) return '';
if (s < 3600) return Math.round(s / 60) + 'm';
if (s < 86400) return (s / 3600).toFixed(1) + 'h';
return (s / 86400).toFixed(1) + 'd';
}
function _renderXfer(x) {
const el = document.getElementById('vv-pt-xfer-body');
if (!el) return;
const peers = (x && x.peers) || {};
const names = Object.keys(peers);
if (!names.length) {
el.innerHTML = `<div style="font-size:11px;color:#444;">No samples yet — the mesh sampler
records once a minute. Figures appear within a minute or two.</div>`;
return;
}
let html = '';
for (const p of names) {
const d = peers[p];
const live = d.live;
html += `<div style="margin-bottom:6px;">
<div style="display:flex;justify-content:space-between;align-items:baseline;gap:8px;
padding-bottom:4px;border-bottom:1px solid #1b1b1b;">
<span style="font-size:11px;color:#888;font-family:monospace;">${vvEscHtml(p)}</span>
<span style="font-size:10px;color:${live && (live.tx_bps + live.rx_bps) > 1024 ? '#4caf50' : '#3a3a3a'};font-family:monospace;">
${live ? '↑ ' + _vvBytes(live.tx_bps) + '/s ↓ ' + _vvBytes(live.rx_bps) + '/s' : 'idle'}
</span>
</div>`;
const rows = [['Last 24 hours','24h'], ['Last 7 days','7d'], ['Last 30 days','30d']];
for (const [label, key] of rows) {
const w = (d.windows || {})[key] || {};
const total = (w.tx || 0) + (w.rx || 0);
// A window whose samples do not span it yet is labelled with what it does cover, rather
// than presenting a partial figure under a full-window heading.
const partial = w.covers && w.covers < ({'24h':86400,'7d':604800,'30d':2592000}[key]) * 0.9;
html += `<div style="display:flex;justify-content:space-between;align-items:baseline;gap:10px;
padding:4px 0;border-bottom:1px solid #151515;">
<span style="color:#666;font-size:11px;">${label}${partial
? ` <span style="color:#3a3a3a;font-size:9px;">only ${_vvDur(w.covers)} recorded</span>` : ''}</span>
<span style="font-family:monospace;font-size:11px;color:${total ? '#4caf50' : '#333'};">
${total ? '↑ ' + _vvBytes(w.tx) + ' ↓ ' + _vvBytes(w.rx) : 'nothing'}
</span></div>`;
}
const s = d.session || {};
html += `<div style="display:flex;justify-content:space-between;align-items:baseline;gap:10px;padding:4px 0;">
<span style="color:#666;font-size:11px;">Since tailscaled start</span>
<span style="font-family:monospace;font-size:11px;color:#7a7a7a;">
↑ ${_vvBytes(s.tx)} ↓ ${_vvBytes(s.rx)}</span>
</div></div>`;
}
el.innerHTML = html;
}
//
// A window with runs but no bytes says "no data moved", not "0.00 GB". They are different
// claims: rsync completing with nothing to send is the normal state for a mirror that is
// already in step, and printing a zero total invites the reading that the sync is broken.
// A window with no runs at all says so separately again.
// ── Config bar ────────────────────────────────────────────────────────────────