Files
Gmer4Lfe 793a75b8a2 Remember the last ten manual syncs so a repeat is one click
Retyping two paths and nine checkboxes correctly every time is where the
mistakes come from, and "one-off" described how a transfer is scheduled rather
than how often it is run. Pinned entries are exempt from the rotation — the
command used twice a year is both the most worth keeping and the first that ten
ordinary runs evict. Stored server-side, so the list is there from any screen,
and loading one fills the form and stops rather than running it.
2026-08-14 17:38:48 -04:00

509 lines
26 KiB
PHP

<?php
// ═══════════════════════════════════════════════════════════════════════════════════════════════
// PURPOSE
// Ad-hoc rsync between hosts. Picks a partner, browses both filesystems to choose a source
// and destination, runs a transfer in the background, streams its output, and cancels it —
// the manual sync panel on the rsync tab.
//
// OPERATIONAL MODEL
// Entirely outside the profile system. The scheduled tiers sync configured shares with
// configured profiles; this is for the one-off move that does not belong in a conf file —
// seeding a new host, recovering a share, copying something once. Nothing here is scheduled.
//
// The last ten are remembered, which the header used to deny. "One-off" turned out to describe
// how a transfer is scheduled, not how often it is run: seeding a host and recovering a share
// are done repeatedly, and retyping two paths and nine checkboxes correctly each time is where
// the mistakes come from. Pinned entries are exempt from the rotation, because the command used
// twice a year is both the most valuable to keep and the first that ten ordinary runs evict.
// Recording is a record of intent only — nothing re-runs itself, and loading an entry fills the
// form and stops.
//
// Eight actions on one URL, in the order the panel uses them: recent, hosts, browse, browse_local,
// run, poll, stop. run returns a token immediately and the transfer continues detached; poll
// reads its log until a sentinel appears; stop kills it.
//
// Progress is a log file in /tmp keyed by token, terminated by a __DONE__ sentinel the
// wrapper appends after rsync exits. That sentinel is what lets poll distinguish "still
// running" from "finished" without inspecting a process — and the pid file is the backstop
// for the case where rsync died without reaching it.
//
// Both filesystems are browsed the same way, one directory at a time. The remote side runs
// the same find over SSH that the local side runs directly, so the two panes behave
// identically.
//
// DESIGN PRINCIPLES
// Source is always local, destination always remote.
// The direction is fixed. A tool that could pull as well as push would need twice the
// path validation and would make "which side am I about to overwrite" a question the
// user has to answer correctly every time.
//
// Trailing slashes are normalised onto both paths.
// rsync's most consequential piece of syntax is whether the source ends in a slash. It
// is appended unconditionally here so the transfer always means "the contents of", never
// "the directory itself nested inside".
//
// One manual sync at a time, host-wide.
// Enforced by scanning for any live pid file, not per token. Two concurrent ad-hoc
// transfers would compete for the same bandwidth the scheduled tiers are also using.
//
// An inconclusive remote check proceeds; a definite failure stops.
// A destination that reports 'missing' is refused. An SSH check that answers neither
// 'ok' nor 'missing' — Tailscale still coming up, for instance — is allowed through,
// because rsync will fail cleanly and visibly in the log if the path really is wrong.
//
// OPERATIONAL SAFEGUARDS
// System paths are refused as both source and destination.
// An explicit blocklist — /, /proc, /sys, /dev, /run, /etc, /bin, /sbin, /usr, /lib,
// /lib64, /boot/EFI, /tmp — compared after trailing-slash normalisation so /etc and
// /etc/ are the same answer. This is the guard against a blank or truncated field
// collapsing into a path that would sync the operating system over the network.
//
// Both paths must be absolute, traversal-free, and free of shell metacharacters.
// Leading slash required, '..' rejected, and null bytes, newlines, backticks and $
// rejected outright — then escapeshellarg() on top.
//
// rsync flags cannot become shell syntax.
// Control characters are rejected and every remaining token is escaped individually.
// The previous blocklist stripped metacharacters but not newlines, and these flags are
// spliced into a `bash -c` script — a newline would have started a second command
// inside it. Escaping each token is right about every character; a blocklist has to be.
//
// The bandwidth limit is cast, not filtered.
// (int) with a max(0, …) floor, so it can only ever be a non-negative number.
//
// The remote user is reduced to a safe character set before it is used.
// preg_replace strips everything outside [a-z0-9_.-], with 'root' as the fallback when
// nothing survives.
//
// The local source is confirmed to exist before anything is launched.
//
// Tokens are random and fixed-length.
// bin2hex(random_bytes(8)) produces the token; poll and stop both reduce the submitted
// value to hex and require exactly 16 characters, so neither can name a file outside
// the /tmp/vv_ms_ namespace. A transfer's log is readable by anyone who can guess its
// token, so the token is not guessable.
//
// Stop escalates and then cleans up.
// SIGTERM, 400ms, SIGKILL — rsync should be given the chance to finish its current file
// and close its connection. The cancellation is then written into the log with the
// sentinel, so a poll already in flight terminates cleanly instead of hanging.
//
// Poll detects a crashed transfer as well as a finished one.
// A missing sentinel with a dead pid is reported done with an explicit note, so a
// transfer killed by OOM or a reboot does not leave the panel polling forever.
//
// SSH is non-interactive and time-boxed.
// BatchMode=yes and ConnectTimeout=10 on the transfer, 8s and 4s on the browse and ping
// calls — nothing here can sit waiting for a password or a dead host.
//
// Log files are removed on the terminal poll, so a completed transfer does not leave its
// output in /tmp indefinitely. /tmp is tmpfs, so an abandoned one clears at reboot anyway.
//
// Accepted exposure: browse can list any directory on either host.
// Directory names only — no file contents, no file names. browse and poll are GET reads
// bounded by the WebGUI session; run and stop are POST and are CSRF-guarded by Unraid's
// auto_prepend before any code here runs. See README-unraid.md.
//
// REQUEST
// GET ?action=hosts partners, Tailscale state, key presence
// GET ?action=browse&host=<slot>&path=/dir remote directory listing over SSH
// GET ?action=browse_local&path=/dir local directory listing
// POST action=run local=/src host=<slot> remote_path=/dst
// [user=root] [bw_limit=<KB/s>] [use_key=0|1] [flags=<rsync flags>]
// GET ?action=poll&token=<hex16> output so far, and whether it finished
// POST action=stop token=<hex16> cancel a running transfer
// GET ?action=recent the remembered syncs, pinned first then newest
// POST action=recent_update id=<hex12> op=rename|sticky|delete
// [name=<label>] [value=0|1] rename, pin/unpin, or forget one entry
//
// RESPONSE
// hosts {"ok":true,"hosts":[{slot,id,hostname,online,ip}],"has_key":bool,"ssh_key":"…"}
// browse {"ok":true,"path","dirs":[…],"parent":…} max 200 entries
// run {"ok":true,"token":"<hex16>"}
// poll {"ok":true,"output":"…","done":bool,"started":bool}
// stop {"ok":true}
// {"ok":false,"error":"Invalid path"|"Missing: …"|"Refusing to sync from system path: …"
// |"Local source does not exist: …"|"Another manual sync is already
// running — stop it first."|"Remote destination does not exist: …"
// |"SSH connection failed to …"|"Invalid flags"|"Invalid token"
// |"Unknown action"}
//
// DEPENDS ON
// include/config.php vv_detect_host(), vv_read_conf_raw(), vv_conf_vars(),
// vv_resolve_tailscale_ip()
// include/arrs.php vv_arr_known_hosts(), vv_arr_scalar()
// include/partnership.php vv_pt_ts_peers(), vv_pt_ssh()
// /tmp/vv_ms_<token>.log|.pid per-transfer state
// ═══════════════════════════════════════════════════════════════════════════════════════════════
header('Content-Type: application/json');
header('Cache-Control: no-store, no-cache');
require_once dirname(__DIR__) . '/include/config.php';
require_once dirname(__DIR__) . '/include/arrs.php';
require_once dirname(__DIR__) . '/include/partnership.php'; // vv_pt_ts_peers, vv_pt_ssh
function _ms_caller(): array {
$host = vv_detect_host();
$id = strtoupper($host);
$raw = vv_read_conf_raw($host . '.conf');
$sshKey = vv_arr_scalar($raw, $id . '_SSH_KEY');
return ['host' => $host, 'ssh_key' => $sshKey];
}
function _ms_target(string $slot, string $sshKey): array {
$vars = vv_conf_vars();
$hostname = $vars[strtoupper($slot)] ?? '';
if (!$hostname) return ['ok' => false, 'error' => 'Unknown host: ' . $slot];
if (!$sshKey || !file_exists($sshKey))
return ['ok' => false, 'error' => 'SSH key not configured on this host'];
$ip = vv_resolve_tailscale_ip($hostname);
if (!$ip) return ['ok' => false, 'error' => 'Cannot reach ' . $hostname . ' via Tailscale'];
return ['ok' => true, 'ip' => $ip, 'hostname' => $hostname];
}
$action = trim($_GET['action'] ?? $_POST['action'] ?? '');
// ── hosts ─────────────────────────────────────────────────────────────────────
if ($action === 'hosts') {
['host' => $current, 'ssh_key' => $sshKey] = _ms_caller();
$all = vv_arr_known_hosts();
$peers = vv_pt_ts_peers();
$out = [];
foreach ($all as $slot => $hostname) {
if ($slot === $current) continue;
$label = strtolower($hostname);
$ts = $peers[$label] ?? [];
$out[] = [
'slot' => $slot,
'id' => strtoupper($slot),
'hostname' => $hostname,
'online' => $ts['online'] ?? null,
'ip' => $ts['ip'] ?? null,
];
}
echo json_encode(['ok' => true, 'hosts' => $out, 'has_key' => !empty($sshKey) && file_exists($sshKey), 'ssh_key' => $sshKey]);
exit;
}
// ── browse remote (SSH) ───────────────────────────────────────────────────────
if ($action === 'browse') {
$slot = trim($_GET['host'] ?? '');
$path = trim($_GET['path'] ?? '/mnt/user');
if (!preg_match('#^/[^\0]*$#', $path) || str_contains($path, '..')) {
echo json_encode(['ok' => false, 'error' => 'Invalid path']); exit;
}
['ssh_key' => $sshKey] = _ms_caller();
$t = _ms_target($slot, $sshKey);
if (!$t['ok']) { echo json_encode($t); exit; }
$clean = rtrim($path, '/') ?: '/';
$out = vv_pt_ssh($t['ip'], $sshKey,
'find ' . escapeshellarg($clean) . ' -maxdepth 1 -mindepth 1 -type d 2>/dev/null',
8);
$dirs = array_values(array_filter(array_map('trim', explode("\n", $out))));
sort($dirs);
$dirs = array_slice($dirs, 0, 200);
// If empty, do a quick SSH echo to distinguish "no dirs" from "SSH failed"
if (empty($dirs)) {
$ping = trim(vv_pt_ssh($t['ip'], $sshKey, 'echo ok', 4));
if ($ping !== 'ok') {
echo json_encode(['ok' => false, 'error' => 'SSH connection failed to ' . $t['hostname']]);
exit;
}
}
$parent = ($clean !== '/') ? (dirname($clean) ?: '/') : null;
echo json_encode(['ok' => true, 'path' => $clean, 'dirs' => $dirs, 'parent' => $parent]);
exit;
}
// ── browse local ──────────────────────────────────────────────────────────────
if ($action === 'browse_local') {
$path = trim($_GET['path'] ?? '/mnt/user');
if (!preg_match('#^/[^\0]*$#', $path) || str_contains($path, '..')) {
echo json_encode(['ok' => false, 'error' => 'Invalid path']); exit;
}
$clean = rtrim($path, '/') ?: '/';
if (!is_dir($clean)) {
echo json_encode(['ok' => false, 'error' => 'Not a directory: ' . $clean]); exit;
}
$out = shell_exec('find ' . escapeshellarg($clean) . ' -maxdepth 1 -mindepth 1 -type d 2>/dev/null | sort | head -200') ?: '';
$dirs = array_values(array_filter(array_map('trim', explode("\n", $out))));
$parent = ($clean !== '/') ? (dirname($clean) ?: '/') : null;
echo json_encode(['ok' => true, 'path' => $clean, 'dirs' => $dirs, 'parent' => $parent]);
exit;
}
// ── run (background, returns token) ──────────────────────────────────────────
if ($_SERVER['REQUEST_METHOD'] === 'POST' && $action === 'run') {
$local = trim($_POST['local'] ?? '');
$slot = trim($_POST['host'] ?? '');
$remotePath = trim($_POST['remote_path'] ?? '');
$user = preg_replace('/[^a-z0-9_.-]/i', '', trim($_POST['user'] ?? 'root')) ?: 'root';
$bwLimit = max(0, (int)($_POST['bw_limit'] ?? 0));
$useKey = ($_POST['use_key'] ?? '1') !== '0';
// Control characters are rejected outright — a newline would start a second command
// inside the bash -c script this is spliced into. What survives is split on whitespace
// and escaped per token below, so the shell never parses any of it as syntax.
$rawFlags = trim($_POST['flags'] ?? '');
if ($rawFlags !== '' && preg_match('/[\x00-\x1f\x7f]/', $rawFlags)) {
echo json_encode(['ok' => false, 'error' => 'Invalid flags']); exit;
}
$flagList = $rawFlags !== ''
? preg_split('/\s+/', $rawFlags, -1, PREG_SPLIT_NO_EMPTY)
: ['-av', '--stats'];
foreach (['local' => $local, 'host' => $slot, 'remote_path' => $remotePath] as $f => $v) {
if (!$v) { echo json_encode(['ok' => false, 'error' => 'Missing: ' . $f]); exit; }
}
foreach ([$local, $remotePath] as $p) {
if (!str_starts_with($p, '/') || str_contains($p, '..') || preg_match('/[\x00\n\r`$]/', $p)) {
echo json_encode(['ok' => false, 'error' => 'Invalid path: ' . $p]); exit;
}
}
// ── Safeguards ────────────────────────────────────────────────────────────
// Block syncing from/to dangerous system paths
$blocked = ['/', '/proc', '/sys', '/dev', '/run', '/etc', '/bin', '/sbin',
'/usr', '/lib', '/lib64', '/boot/EFI', '/tmp'];
foreach ($blocked as $b) {
if (rtrim($local, '/') === $b) {
echo json_encode(['ok' => false, 'error' => 'Refusing to sync from system path: ' . $b]); exit;
}
if (rtrim($remotePath, '/') === $b) {
echo json_encode(['ok' => false, 'error' => 'Refusing to sync to system path: ' . $b]); exit;
}
}
// Local source must exist
if (!file_exists($local)) {
echo json_encode(['ok' => false, 'error' => 'Local source does not exist: ' . $local]); exit;
}
// Prevent concurrent manual syncs
foreach (glob('/tmp/vv_ms_*.pid') ?: [] as $pf) {
$pid = (int)trim(@file_get_contents($pf) ?: '0');
if ($pid > 0 && file_exists('/proc/' . $pid)) {
echo json_encode(['ok' => false, 'error' => 'Another manual sync is already running — stop it first.']); exit;
}
@unlink($pf); // stale
}
['ssh_key' => $sshKey] = _ms_caller();
$t = _ms_target($slot, $sshKey);
if (!$t['ok']) { echo json_encode($t); exit; }
// Remote destination must exist
$destCheck = trim(vv_pt_ssh($t['ip'], $sshKey,
'test -d ' . escapeshellarg($remotePath) . ' && echo ok || echo missing', 5));
if ($destCheck === 'missing') {
echo json_encode(['ok' => false, 'error' => 'Remote destination does not exist: ' . $remotePath]); exit;
}
if ($destCheck !== 'ok') {
// SSH check inconclusive — log warning but proceed; rsync will fail cleanly if needed
// (e.g. Tailscale not yet up, rsync error will surface in output)
}
// ── Build and launch ──────────────────────────────────────────────────────
$token = bin2hex(random_bytes(8));
$logFile = '/tmp/vv_ms_' . $token . '.log';
$pidFile = '/tmp/vv_ms_' . $token . '.pid';
if ($useKey && $sshKey && file_exists($sshKey)) {
$sshOpts = 'ssh -i ' . escapeshellarg($sshKey) . ' -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=10';
} else {
$sshOpts = 'ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10';
}
if ($bwLimit) $flagList[] = '--bwlimit=' . (int)$bwLimit;
$flags = implode(' ', array_map('escapeshellarg', $flagList));
$src = escapeshellarg(rtrim($local, '/') . '/');
$dst = escapeshellarg($user . '@' . $t['ip'] . ':' . rtrim($remotePath, '/') . '/');
// Start rsync in background, capture its PID for stop support
$inner = "rsync $flags -e " . escapeshellarg($sshOpts) . " $src $dst >> " . escapeshellarg($logFile) . " 2>&1 &"
. " RSYNC_PID=\$!;"
. " echo \$RSYNC_PID > " . escapeshellarg($pidFile) . ";"
. " wait \$RSYNC_PID;"
. " echo __DONE__ >> " . escapeshellarg($logFile) . ";"
. " rm -f " . escapeshellarg($pidFile);
shell_exec('nohup bash -c ' . escapeshellarg($inner) . ' &>/dev/null &');
// Recorded at launch, not at completion. What is worth recalling is the command that was
// assembled — a transfer that failed is often precisely the one to run again, and a poll that
// never returns because the tab was closed would otherwise lose it entirely.
//
// Flags are stored without --bwlimit: it was appended above from its own field, and keeping it
// in the list would restore it into the flag checkboxes where there is no such checkbox.
$recent = vv_ms_recent_load();
$key = vv_ms_recent_key($local, $slot, $remotePath);
$entry = [
'id' => $key,
'ts' => time(),
'local' => $local,
'slot' => $slot,
'user' => $user,
'rpath' => $remotePath,
'flags' => array_values(array_filter($flagList, fn($f) => !str_starts_with($f, '--bwlimit='))),
'bw' => $bwLimit,
'use_key' => $useKey,
'name' => '',
'sticky' => false,
];
$found = false;
foreach ($recent as $i => $r) {
if (($r['id'] ?? '') !== $key) continue;
// The same copy run again: refresh everything except what the operator chose about it.
// A name and a pin are decisions about the entry, not properties of the last run.
$entry['name'] = $r['name'] ?? '';
$entry['sticky'] = !empty($r['sticky']);
$recent[$i] = $entry;
$found = true;
break;
}
if (!$found) $recent[] = $entry;
vv_ms_recent_save($recent);
echo json_encode(['ok' => true, 'token' => $token]);
exit;
}
// ── stop ──────────────────────────────────────────────────────────────────────
if ($_SERVER['REQUEST_METHOD'] === 'POST' && $action === 'stop') {
$token = preg_replace('/[^a-f0-9]/', '', trim($_POST['token'] ?? ''));
if (!$token || strlen($token) !== 16) {
echo json_encode(['ok' => false, 'error' => 'Invalid token']); exit;
}
$pidFile = '/tmp/vv_ms_' . $token . '.pid';
$logFile = '/tmp/vv_ms_' . $token . '.log';
$pid = (int)trim(@file_get_contents($pidFile) ?: '0');
if ($pid > 0 && file_exists('/proc/' . $pid)) {
shell_exec('kill -TERM ' . $pid . ' 2>/dev/null');
usleep(400000); // 400ms grace
if (file_exists('/proc/' . $pid)) shell_exec('kill -KILL ' . $pid . ' 2>/dev/null');
}
@unlink($pidFile);
@file_put_contents($logFile, "\n\n--- Cancelled by user ---\n__DONE__\n", FILE_APPEND);
echo json_encode(['ok' => true]);
exit;
}
// ── poll ──────────────────────────────────────────────────────────────────────
if ($action === 'poll') {
$token = preg_replace('/[^a-f0-9]/', '', trim($_GET['token'] ?? ''));
if (!$token || strlen($token) !== 16) {
echo json_encode(['ok' => false, 'error' => 'Invalid token']); exit;
}
$logFile = '/tmp/vv_ms_' . $token . '.log';
$pidFile = '/tmp/vv_ms_' . $token . '.pid';
if (!file_exists($logFile)) {
echo json_encode(['ok' => true, 'output' => '', 'done' => false, 'started' => false]); exit;
}
$content = file_get_contents($logFile) ?: '';
$done = str_contains($content, '__DONE__');
if ($done) {
$content = str_replace(['__DONE__', "\n\n\n"], ['', "\n\n"], $content);
@unlink($logFile);
@unlink($pidFile);
}
// Check if rsync process is actually alive (catches crashes without __DONE__)
$pid = (int)trim(@file_get_contents($pidFile) ?: '0');
if (!$done && $pid > 0 && !file_exists('/proc/' . $pid)) {
$done = true;
$content .= "\n\n--- Process ended unexpectedly ---";
@unlink($pidFile);
}
echo json_encode(['ok' => true, 'output' => $content, 'done' => $done, 'started' => true]);
exit;
}
// ── recent ────────────────────────────────────────────────────────────────────
// The last handful of manual syncs, so a repeat is one click rather than retyping two paths and
// nine checkboxes correctly. Kept on the server rather than in localStorage: this operator drives
// the same install from two desktop screens and a phone, and a recall list that only exists in
// the browser that made it is a recall list that is missing whenever it is wanted.
//
// Rotation is by recency with a hard cap, except for pinned rows, which never rotate out and do
// not count against the cap. That is the whole reason pinning exists — a seeding command used
// twice a year is exactly the one worth keeping and exactly the one ten ordinary runs would push
// off the end.
//
// Identity is the transfer itself — source, host, destination — not the flags. Re-running the same
// copy with --delete added is the same entry with different options, and keeping both would fill
// the list with near-duplicates that differ in the one place nobody reads.
function vv_ms_recent_path(): string {
return rtrim((string)(vv_conf_vars()['STATE_DIR'] ?? STATE_DIR), '/') . '/manual_sync_recent.json';
}
function vv_ms_recent_load(): array {
$raw = @file_get_contents(vv_ms_recent_path());
if ($raw === false) return [];
$d = json_decode($raw, true);
return is_array($d) ? $d : [];
}
function vv_ms_recent_save(array $rows): bool {
// Pinned first so the cap can never evict one, then newest, then trimmed.
usort($rows, function ($a, $b) {
$p = (int)!empty($b['sticky']) <=> (int)!empty($a['sticky']);
return $p !== 0 ? $p : ((int)($b['ts'] ?? 0) <=> (int)($a['ts'] ?? 0));
});
$kept = [];
$loose = 0;
foreach ($rows as $r) {
if (!empty($r['sticky'])) { $kept[] = $r; continue; }
if ($loose >= 10) continue;
$loose++;
$kept[] = $r;
}
$path = vv_ms_recent_path();
$tmp = $path . '.tmp';
if (@file_put_contents($tmp, json_encode($kept, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES)) === false) {
return false;
}
return @rename($tmp, $path);
}
// Identity of a transfer, for de-duplicating repeats of the same copy.
function vv_ms_recent_key(string $local, string $slot, string $rpath): string {
return substr(hash('sha256', $local . '|' . $slot . '|' . $rpath), 0, 12);
}
if ($action === 'recent') {
echo json_encode(['ok' => true, 'rows' => array_values(vv_ms_recent_load())]);
exit;
}
if ($_SERVER['REQUEST_METHOD'] === 'POST' && $action === 'recent_update') {
$id = preg_replace('/[^a-f0-9]/', '', trim($_POST['id'] ?? ''));
$op = trim($_POST['op'] ?? '');
if ($id === '') { echo json_encode(['ok' => false, 'error' => 'Missing id']); exit; }
$rows = vv_ms_recent_load();
$hit = false;
foreach ($rows as $i => $r) {
if (($r['id'] ?? '') !== $id) continue;
$hit = true;
if ($op === 'rename') {
// Plain text only, and short. It is a label in a list, and anything richer is markup
// waiting to be rendered somewhere that forgot to escape it.
$name = trim((string)($_POST['name'] ?? ''));
$name = preg_replace('/[^\p{L}\p{N} ._\-\/→>]/u', '', $name);
$rows[$i]['name'] = mb_substr($name, 0, 60);
} elseif ($op === 'sticky') {
$rows[$i]['sticky'] = ($_POST['value'] ?? '0') === '1';
} elseif ($op === 'delete') {
unset($rows[$i]);
} else {
echo json_encode(['ok' => false, 'error' => 'Unknown op']); exit;
}
break;
}
if (!$hit) { echo json_encode(['ok' => false, 'error' => 'No such entry']); exit; }
echo json_encode(['ok' => vv_ms_recent_save(array_values($rows))]);
exit;
}
echo json_encode(['ok' => false, 'error' => 'Unknown action']);