Drop subreddit front pages from degoog results
Its Reddit engine returns a community's landing page for any query at all — a question about ZFS scrubs came back with Rocket League Esports and Dividend Investing, interleaved one in two, so half of what reached the model was noise. Threads from that same engine are frequently the best result there is, so the engine stays and only the URL shape goes: /r/<name> with nothing after it is a front page by definition. That is a fact about the URL, not an opinion about how well it matches, and this file does no similarity scoring of its own.
This commit is contained in:
@@ -175,11 +175,26 @@ function vv_ai_web_degoog(string $q): ?array {
|
||||
if ($base === '') return null;
|
||||
$d = vv_ai_web_http($base . '/api/search?q=' . rawurlencode($q));
|
||||
if ($d === null) return null;
|
||||
return array_map(fn($r) => ['title' => $r['title'] ?? '',
|
||||
'url' => $r['url'] ?? '',
|
||||
'snippet' => ($r['snippet'] ?? '') !== '' ? $r['snippet']
|
||||
: ($r['content'] ?? '')],
|
||||
array_slice($d['results'] ?? [], 0, vv_ai_web_results_max()));
|
||||
|
||||
// Its Reddit engine returns subreddit front pages for any query at all — "Rocket League
|
||||
// Esports" and "Dividend Investing" came back for a question about ZFS scrubs, interleaved
|
||||
// one in two, so half of what reached the model was noise. Reddit *threads* from the same
|
||||
// engine are often the best result there is, so the engine is not excluded; the shape of URL
|
||||
// that cannot contain an answer is.
|
||||
//
|
||||
// Structural, not a relevance judgement. /r/<name> with nothing after it is a community's
|
||||
// front page by definition, which is a fact about the URL rather than an opinion about how
|
||||
// well it matches — this file does no similarity scoring of its own and should not start.
|
||||
$rows = [];
|
||||
foreach ($d['results'] ?? [] as $r) {
|
||||
$url = (string)($r['url'] ?? '');
|
||||
if (preg_match('#^https?://(?:[a-z0-9-]+\.)*reddit\.com/r/[^/?#]+/?$#i', $url)) continue;
|
||||
$rows[] = ['title' => $r['title'] ?? '',
|
||||
'url' => $url,
|
||||
'snippet' => ($r['snippet'] ?? '') !== '' ? $r['snippet'] : ($r['content'] ?? '')];
|
||||
if (count($rows) >= vv_ai_web_results_max()) break;
|
||||
}
|
||||
return $rows;
|
||||
}
|
||||
|
||||
function vv_ai_web_searxng(string $q): ?array {
|
||||
|
||||
Reference in New Issue
Block a user