Gmer4Lfe
df3c4bb369
A job killed hard never corrects its own record, so 'running' outlives the process and the pid can since belong to something else; array-start jobs get setsid too, so Stop can reach past their direct children.
232 lines
12 KiB
PHP
232 lines
12 KiB
PHP
<?php
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
// PURPOSE
|
|
// Job killer. Terminates a running job, clears the lock files it left behind, and corrects
|
|
// its stat file — the stop button on the scheduler page.
|
|
//
|
|
// OPERATIONAL MODEL
|
|
// Escalating termination, then cleanup. SIGTERM to the whole process group, up to three
|
|
// seconds to exit, SIGKILL if it did not. The group is the target rather than the pid
|
|
// because a Varaverk script is mostly other processes — rsync, ssh, curl, docker — and
|
|
// killing only the parent would orphan every one of them still holding the resources the
|
|
// next run needs.
|
|
//
|
|
// The stat file is the job's own record, and a killed job never gets to correct it. This
|
|
// endpoint does that on its behalf, which is what stops the scheduler page from showing a
|
|
// spinner for a job that no longer exists.
|
|
//
|
|
// DESIGN PRINCIPLES
|
|
// Kills the group only when the job owns the group; otherwise the tree.
|
|
// `ps -o pgid=` resolves the process group, and it is used only if it equals the recorded
|
|
// pid — i.e. the job is the group leader. Otherwise the fallback is pkill -P plus the pid
|
|
// itself. Two strategies, because a job whose runner already exited can leave children
|
|
// whose group id is no longer discoverable from the recorded pid, and because a job that
|
|
// was not started under setsid shares its group with whatever launched it.
|
|
//
|
|
// Waits before escalating.
|
|
// Six 500ms checks between TERM and KILL. Scripts have cleanup handlers — releasing
|
|
// locks, finishing a write, unmounting — and killing immediately would skip exactly the
|
|
// work that makes the next run safe.
|
|
//
|
|
// Reports honestly when the kill failed.
|
|
// ok mirrors whether the process is actually gone. A process in uninterruptible sleep
|
|
// survives SIGKILL, and the response says so rather than claiming success and leaving
|
|
// the user to discover the job is still running.
|
|
//
|
|
// OPERATIONAL SAFEGUARDS
|
|
// The job id is validated, and the pid is an integer before it reaches a shell.
|
|
// ^[a-zA-Z0-9_./\-]+\.sh$ with an explicit '..' check on the id, and (int) casts on
|
|
// every pid — the recorded one and each lock's owner — before interpolation. The signal
|
|
// commands are the only shell in this file and none of them can carry anything but a
|
|
// number.
|
|
//
|
|
// pid < 2 is refused.
|
|
// A zero, missing, or malformed pid in the stat file would make `kill -TERM -0` signal
|
|
// the caller's own process group — the web server. Rejecting anything below 2 also
|
|
// excludes init.
|
|
//
|
|
// A group is never signalled unless the job leads it.
|
|
// pgid must equal the recorded pid, and must not equal this process's own group. This is
|
|
// the guard the pid < 2 check was mistaken for: pid was always a valid, live number, and
|
|
// the group it named was still the web server's. On 2026-08-07 stopping a genuinely
|
|
// running transcode_management.sh SIGTERMed then SIGKILLed the php-fpm group and took the
|
|
// WebGUI down; Docker was unaffected, so it presented as an OS crash. The failure needs a
|
|
// job that is still alive when Stop is pressed, which is why earlier stop tests — where
|
|
// the script had already exited and `ps` returned nothing — passed.
|
|
//
|
|
// A job that is not running is a no-op success.
|
|
// Three paths exit before any signal is sent: no stat file, status not 'running', and a
|
|
// recorded pid with no /proc entry. The third is the one that matters, because the other
|
|
// two only cover a job that got to write its own ending. run_job.sh updates the stat file
|
|
// on exit, but SIGKILL, an OOM kill and a power loss all skip that — leaving 'running'
|
|
// and a pid the kernel is free to hand to something else. Signalling on the strength of
|
|
// that file alone is signalling a stranger. run.php and status.php already refuse to
|
|
// believe a 'running' stat without the /proc check; this endpoint is the one that sends
|
|
// SIGKILL, so it is the last place that should have been taking the file's word for it.
|
|
//
|
|
// The stale entry is corrected to 'error' on the way out rather than left alone: the job
|
|
// did not stop, it died, and a status of 'stopped' would claim this endpoint did
|
|
// something it did not do.
|
|
//
|
|
// What this does not prove is that a live /proc/<pid> is still *this* job's process —
|
|
// a pid reissued to something unrelated between the stat write and the click looks
|
|
// identical. Closing that needs the process start-time out of /proc/<pid>/stat compared
|
|
// against the recorded start. Not done here, and deliberately: it is real machinery, and
|
|
// the window it closes is far narrower than the one above.
|
|
//
|
|
// The pid is only cleared from the stat file if the process is confirmed gone.
|
|
// D-state processes survive SIGKILL. Clearing the pid there would lose the only handle
|
|
// anyone has on a process that is still holding locks, and would report the job stopped
|
|
// while it continues to run.
|
|
//
|
|
// Lock clearing is bounded to the lock directory and to dead owners.
|
|
// glob over /tmp/unraid_locks/*.lock — a hardcoded literal, not a config value — and a
|
|
// lock is removed only when its recorded owner is this pid or is no longer in /proc. A
|
|
// lock held by a live, unrelated process is never touched.
|
|
//
|
|
// Deliberate side effect: stale locks from other jobs are reaped too.
|
|
// The dead-owner test is not scoped to this job, so one stop clears every abandoned
|
|
// lock on the host. That is intentional — a lock whose owner does not exist is by
|
|
// definition stale, and leaving it to be found later means a future run refuses to
|
|
// start for no reason.
|
|
//
|
|
// The name-based fallback covers pid rotation.
|
|
// A lock file named after the script is removed regardless of its recorded owner,
|
|
// because a rotated pid can make a genuinely stale lock look live.
|
|
//
|
|
// REQUEST
|
|
// POST id=<Category/name.sh>
|
|
//
|
|
// RESPONSE
|
|
// {"ok":true,"killed":true,"locks":["…"],"error":null}
|
|
// {"ok":true,"msg":"Not running"}
|
|
// {"ok":true,"msg":"Not running — stale pid cleared"} stat said running, process was gone
|
|
// {"ok":false,"killed":false,"locks":[…],
|
|
// "error":"Process still alive after SIGKILL (D-state) — lock may persist"}
|
|
// {"ok":false,"error":"Invalid id"|"No stat file — script may not be running"
|
|
// |"No valid PID in stat file"}
|
|
//
|
|
// DEPENDS ON
|
|
// include/scheduler.php vv_job_stat_path()
|
|
// /tmp/unraid_locks lock files written by common.sh's locking helper
|
|
// ═══════════════════════════════════════════════════════════════════════════════════════════════
|
|
header('Content-Type: application/json');
|
|
require_once dirname(__DIR__) . '/include/scheduler.php';
|
|
|
|
$id = trim($_POST['id'] ?? '');
|
|
|
|
if (!$id || !preg_match('/^[a-zA-Z0-9_.\/\-]+\.sh$/', $id) || str_contains($id, '..')) {
|
|
echo json_encode(['ok' => false, 'error' => 'Invalid id']);
|
|
exit;
|
|
}
|
|
|
|
$statFile = vv_job_stat_path($id);
|
|
if (!file_exists($statFile)) {
|
|
echo json_encode(['ok' => false, 'error' => 'No stat file — script may not be running']);
|
|
exit;
|
|
}
|
|
|
|
$stat = json_decode(file_get_contents($statFile) ?: '{}', true) ?: [];
|
|
|
|
if (($stat['status'] ?? '') !== 'running') {
|
|
echo json_encode(['ok' => true, 'msg' => 'Not running']);
|
|
exit;
|
|
}
|
|
|
|
$pid = (int)($stat['pid'] ?? 0);
|
|
if ($pid < 2) {
|
|
echo json_encode(['ok' => false, 'error' => 'No valid PID in stat file']);
|
|
exit;
|
|
}
|
|
|
|
// The stat file says 'running'; the process table is what decides whether that is still true.
|
|
// A job killed hard never got to correct its own file, so 'running' can outlive the process by
|
|
// days and the pid can since have been reissued. Correct the record and stop — nothing here is
|
|
// worth signalling.
|
|
if (!file_exists("/proc/$pid")) {
|
|
$stat['status'] = 'error';
|
|
$stat['end'] = $stat['end'] ?? time();
|
|
unset($stat['pid']);
|
|
file_put_contents($statFile, json_encode($stat));
|
|
echo json_encode(['ok' => true, 'msg' => 'Not running — stale pid cleared']);
|
|
exit;
|
|
}
|
|
|
|
// Kill the whole process group so the script and all its children die together — but only when
|
|
// the job genuinely owns that group.
|
|
//
|
|
// The old code took whatever `ps -o pgid=` returned and signalled it. The header claimed the pgid
|
|
// "is usually the same as the session leader PID from run_job.sh", and that was never true: neither
|
|
// `nohup` nor `&` starts a new process group, and PHP's exec() runs the command under a
|
|
// non-interactive `sh -c` where job control is off. The job therefore inherited the process group
|
|
// of the php-fpm worker that spawned it, and `kill -TERM -$pgid` signalled the entire web server
|
|
// group. Pressing Stop on a job that was actually still running took the WebGUI down with it;
|
|
// Docker lives in its own sessions and kept serving, which is what made it look like an OS crash
|
|
// rather than a plugin bug. run.php now prepends setsid so the job is its own leader, and this
|
|
// check is what holds even if it ever stops doing that.
|
|
$pgid = (int)trim(shell_exec("ps -o pgid= -p $pid 2>/dev/null") ?: '0');
|
|
$ownPgid = function_exists('posix_getpgrp') ? (int)posix_getpgrp() : 0;
|
|
|
|
// A group kill is only ever safe when the recorded pid IS the group leader. Anything else means
|
|
// the group is shared with processes this endpoint knows nothing about.
|
|
$groupSafe = ($pgid > 1 && $pgid === $pid && $pgid !== $ownPgid);
|
|
|
|
if ($groupSafe) {
|
|
shell_exec("kill -TERM -$pgid 2>/dev/null");
|
|
} else {
|
|
// Not our group — signal only the process and its direct children.
|
|
shell_exec("pkill -TERM -P $pid 2>/dev/null");
|
|
shell_exec("kill -TERM $pid 2>/dev/null");
|
|
}
|
|
|
|
// Give it up to 3s to exit gracefully
|
|
$dead = false;
|
|
for ($i = 0; $i < 6; $i++) {
|
|
usleep(500000);
|
|
if (!file_exists("/proc/$pid")) { $dead = true; break; }
|
|
}
|
|
|
|
// Force-kill if still alive
|
|
if (!$dead) {
|
|
if ($groupSafe) shell_exec("kill -KILL -$pgid 2>/dev/null");
|
|
shell_exec("pkill -KILL -P $pid 2>/dev/null");
|
|
shell_exec("kill -KILL $pid 2>/dev/null");
|
|
usleep(300000);
|
|
$dead = !file_exists("/proc/$pid");
|
|
}
|
|
|
|
// Clear any lock files in /tmp/unraid_locks whose content matches this PID
|
|
$lockDir = '/tmp/unraid_locks';
|
|
$cleared = [];
|
|
foreach (glob("$lockDir/*.lock") ?: [] as $lf) {
|
|
$content = trim(file_get_contents($lf) ?: '');
|
|
$lockPid = (int)explode(':', $content)[0];
|
|
if ($lockPid === $pid || !file_exists("/proc/$lockPid")) {
|
|
@unlink($lf);
|
|
$cleared[] = basename($lf);
|
|
}
|
|
}
|
|
|
|
// Also clear by script name in case PID rotated
|
|
$scriptBase = basename($id, '.sh');
|
|
$namedLock = "$lockDir/{$scriptBase}.lock";
|
|
if (file_exists($namedLock)) {
|
|
@unlink($namedLock);
|
|
if (!in_array(basename($namedLock), $cleared)) $cleared[] = basename($namedLock);
|
|
}
|
|
|
|
// Update stat file — only clear pid if actually dead (D-state processes survive SIGKILL)
|
|
$now = time();
|
|
$stat['status'] = $dead ? 'stopped' : 'running';
|
|
$stat['end'] = $dead ? $now : ($stat['end'] ?? null);
|
|
$stat['exit'] = $dead ? -1 : ($stat['exit'] ?? null);
|
|
if ($dead) unset($stat['pid']);
|
|
file_put_contents($statFile, json_encode($stat));
|
|
|
|
echo json_encode([
|
|
'ok' => $dead,
|
|
'killed' => $dead,
|
|
'locks' => $cleared,
|
|
'error' => $dead ? null : 'Process still alive after SIGKILL (D-state) — lock may persist',
|
|
]);
|