Here is the methodology and background for how the numbers on the results page were made, the decisions motivating the experimental design, where the questions came from, what the two ratings mean and what was held identical between providers.
I took what I saw from run 1 to create a set of 204 questions: 96 ordinary, 96 deliberately hard, 12 with no answer anywhere, designed to baseline false positives. Of course, there is a bias for the questions created based on the ones I picked.
They spread across six kinds of task: breaking news, how-to, product lookup, local and geographic, recent releases, and live numbers. Additionally, these spanned six subject areas, weighted toward software and everyday questions because that is what I actually point a retrieval agent at.
Every question carries a note written before the run saying what a correct page contains and what a plausible wrong one looks like, the judges never see those notes. They exist so that when a rating looks wrong in the explorer, there is a written intent to argue why.
Four factors tend to make questions harder for search providers to serve.
Trap: a plausible wrong answer sits close to the right one, an older figure or a superseded draft.
Buried: the answer exists but not on the first result.
Multi: no single page has all of it.
Obscure: the subject is thinly covered, so the index either has it or does not.
The point of the split is that easy questions make providers look alike, but as questions get harder, the gaps between providers come up. A question your model could nearly answer unaided does not test retrieval much, and if a whole benchmark is built from those, every provider passes.
The point behind creating this unanswerable set was to create a group of questions to test how far these search providers try to cheat with results that can cause hallucinations for models and agents. Each of these asks about something that sounds entirely real and does not exist.
Correct behaviour is that every page rates 0 or 1. A page rated 2 or 3 means either a provider surfaced something that invented the answer or a judge invented the relevance.
Run 1 asked 48 questions of four providers and graded the answer a model wrote from the results that were returned. Then it checked whether source quality and answer quality moved together. However, in that run there was only a correlation of 0.16, where 1 is lockstep and 0 is unrelated. The model was carrying most of the result.
So run 2 grades the sources instead, doubles the field to eight providers, and uses three judges from three different labs instead of one, so that no single model's taste skews the ranking. It also gives the slow arms room to finish rather than timing them out.
One call to a provider for one query comes back with a set of 8 pages, and that set is the unit everything here is measured on. Each of those pages gets read and rated on its own, and then the 8 of them get read again together as a set, so a single question against a single provider produces 8 page ratings and one set rating from every judge.
The two ratings answer different questions of the same results, both on a 0 to 3 scale. The page rating asks whether one page helps answer one question, and the judge scoring it is never told which provider returned that page, never shown an answer and never asked to write one. The set rating asks whether the whole set is enough to answer at all, and that is the one carrying the ranking on the results page.
These are separate judge calls rather than one derived from the other, so the set rating is never an average of the page ratings. It could not be, because 8 pages each holding a third of the answer and 8 pages holding nothing average out to roughly the same score, and only one of those sets actually lets you answer the question.
Each judge scores on its own and nothing gets merged before it is recorded, so every individual verdict sits in the explorer with the sentence that judge wrote for it. The averaging happens once, at the end, and only so there is something to put in a table: a provider's set rating is the mean of its scores across all 204 questions and all three judges.
Both scales come with a written definition of every score rather than a bare number, because two models can agree on what a 2 means but a naked scale is something each one invents its own version of.
Printed from the file the run imported them from, at the commit these numbers were judged against, so what is on this page cannot drift from what the judges were actually asked. The scale above appears inside each one because that is how a judge sees it.
You judge whether ONE web page helps answer ONE question. Grade the page on this scale:
0 = not relevant: does not address the question
1 = marginal: on the topic, but would not help answer it
2 = relevant: contains part of the answer, or usable evidence toward it
3 = highly relevant: answers the question directly
A page can contain every word of the question and answer nothing — judge whether it answers, not whether it matches keywords. Judge the body, not the headline: if a page's title or opening promises the subject but the actual content is about something else, that is 0, however closely the title matches. A missing publish date is not itself disqualifying; stale content that the question implicitly needs fresh is.
Return JSON only: {"rung": <int 0-3>, "rationale": "<one sentence>"}.
You judge whether ONE web-search provider's WHOLE set of returned sources is enough to answer ONE question. The question you are answering is: could someone answer it from this set alone, without searching again? Grade the set on this scale:
0 = nothing: no source here addresses the question
1 = on topic only: about the right subject, but reading all of it would not answer the question
2 = partial: part of the answer is here; the rest would need another search
3 = answerable: the question can be answered from this set alone
Grade what the sources say, not what they are about. Authoritative, on-topic, cleanly extracted pages that do not contain the answer are a 1 — a set of the right organisation's pages that never states the fact asked for is a failed retrieval, not a good one. If nothing in the set answers the question, say so even when every source is credible.
Judge the body, not the headline: a page whose title promises the subject while its content is about something else contributes nothing. A missing publish date is not itself disqualifying; stale content that the question implicitly needs fresh is.
Return JSON only: {"score": <int 0-3>, "rationale": "<one sentence>"}.
The set prompt replaced an earlier one partway through the run. That one graded whether sources looked good, fresh and authoritative and cleanly extracted, and on a question with no answer anywhere it handed out a passing score while the page judge was correctly rating every page 0 or 1. The sources were good, they just could not answer, and everything published here was scored with the replacement.
One question goes to all eight providers and each returns up to 8 results, and nothing else changes between them: same prompt, same judges, same number of results asked for, caching off.
Of the eight arms, six finished inside one day, but Tavily and Bright Data hit their own rate limits and finished in the early hours of the next morning, which is the one place the arms were not treated identically, since a page that changed overnight would have looked different to those two.
Every setting sits in one file, core/controls.ts, and every stage of the run reads from it, so there is only ever one value of anything for a stage to find. Each provider was set to the best tier it sells, and wherever there was a depth or content knob it got turned up.
| Provider | Product | Content setting | What comes back |
|---|---|---|---|
| Perplexity | Search API /search | search_context_size: high | Excerpts, their maximum; there is no full-page mode |
| Brave | Search API, web endpoint | extra_snippets: true | Snippets only, by architecture; never a page body |
| Parallel | Search API, advanced tier | mode: advanced, 6,000-char excerpts | Dense excerpts; cannot be turned off |
| Exa | /search with inline contents | contents: { text: true } | Page text at Exa's own default length |
| Tavily | Search API /search | search_depth: advanced, include_raw_content: markdown | Full page markdown |
| Serper | Google Search API /search | none; their scrape endpoint was not called | Links and snippets only. The one deliberate downgrade |
| Firecrawl | v2 /search with scrape | scrapeOptions: { formats: ["markdown"] } | Full page markdown |
| Bright Data | SERP API + Web Unlocker | data_format: markdown, via a second paid product | Full page markdown, on the 47% of URLs the unlocker returned |
Serper is the one deliberate downgrade. It sells a separate scrape endpoint and this run does not call it, which I decided before the run rather than after seeing the scores. Bright Data is the only provider that needs a second paid product to get page text at all, and its extraction step missed roughly half the URLs it tried.
Providers return the same pages constantly, so every page from every arm collapses into one pool keyed by normalised URL and gets graded once, and that rating then counts for each provider that returned it. 66.8% of the 7,496 pages came from exactly one provider and 24 came from all eight.
Pooling also blinds the judge, which is a real argument for doing it, since a page is detached from whoever returned it before anyone grades it and there is no provider name left to be swayed by. However, it was a cost decision first and it costs something real: where several arms returned the same URL, the longest extraction anyone returned is the one that got graded, so extraction quality is averaged away on those pages. With a bigger budget I would not pool.
Where the real value probably sits, given that 204 questions is a sample rather than every question anyone could ask. If two of them overlap, the order between those two providers is not a finding.
Of everything that moves in the ratings, how much each thing explains. Which provider you used explains 23.1% of it, and which judge you happened to pick explains 2.2%.
Comparing two providers over the same small set of questions by counting wins, then asking how often a split that lopsided would come up by luck. I used it on the 12 unanswerable questions, where eight providers cannot be ranked but two of them can still be compared.
Each source gets a 1600-character slice, which is what an agent working inside a real context budget would give it, but a bigger slice would let the providers that return whole documents show more of what they extract and it could reorder the bottom of the table. Pooling costs the same measurement a second way, because on the 33% of pages that several arms returned, the longest extraction anyone returned is the one that got graded. Measuring extraction properly means holding the URLs fixed and varying only the extractor, and I have not run that.
Nothing here grades an answer, on purpose. If that is what you want to know then an answer-graded benchmark measures it, bearing in mind that such a number is also grading your model.
Every arm ran once, at whatever concurrency it tolerated, so nothing here measures throughput, quota behaviour, or what any of it costs at volume. What the run did collide with is written up in notes from the run, a provider that throttles at one request, a plan that ran out mid-way, a day that ate most of a month's credits, but those happened once each and none of them is a measurement.
They are written rather than collected from real traffic, and I wrote all of them, so the harness exists in order that you can run your own instead. That is the only real answer to this one.
This is one run, fetched 17–18 August 2026, and every provider is a moving target: indexes change, tiers change, and the model behind one of these can be swapped without anyone telling you. Nothing here is measured twice, so none of these numbers carries an estimate of how far it would move on a re-run, and the confidence intervals only describe sampling across questions inside this one run, which is a narrower claim than it looks.
Three language models grade every page, and which of the three you happen to be reading explains only 2.2% of everything that moves in the ratings, so they are not meaningfully arguing with each other about the ranking. However, that is agreement among themselves rather than agreement with a person. They were calibrated against human-settled pairs before the run, but three models trained on overlapping text can share a blind spot and agree anyway, and the 12 unanswerable questions show exactly that happening, because a plausible near-miss still gets past some of them.
The table has a median latency column so timing is measured, but it is measured from one machine on one network over two days, and a provider that is quick from my connection may not be quick from your region. Take the ordering seriously and the absolute numbers loosely.