{"author": "Ashita Orbis", "category": "lab-notes", "conversationExcerpts": false, "date": "2026-08-15", "description": "104 runs across two identical realizations of a retrieval benchmark with a deterministic scorer: Claude Opus 5, GPT-5.6 Sol, Grok 4.6 and DeepSeek V4-Pro all retrieve, and separate on citation validity and reproducibility instead.", "draft": false, "meansEndsRatio": 0.3, "projects": [], "slug": "078-everyone-found-the-answer", "subtitle": "Four models on the same retrieval task scored 0.743 to 0.533, and the number that would actually decide a deployment is not the score: it is whether the quotes they hand back are real.", "tags": ["lab-notes", "benchmarking", "retrieval", "citations", "measurement"], "title": "Everyone Found the Answer"}
---
A retrieval benchmark usually reports one number per model, and that number cannot see the
difference between an answer that is right and an answer that is right while citing a quote the
source document does not contain. Scoring four models on a task that required both, with the
grading done by a frozen deterministic scorer rather than by a judge model, put that gap on a
column of its own.

Thirteen scoped packets over a frozen document corpus, two local tools, and a strict output
contract: enumerate every in-scope entity, and for each one emit a verbatim excerpt of fifty words
or fewer containing the entity name, as one JSON array and nothing else. Two identical
realizations, four arms, **104 worker runs, no failures and no excluded cells**. Each arm's model
identity was captured from its harness's own telemetry rather than from the flag it was launched
with, and the transports were probed before the sweep and again after: Claude Opus 5 and GPT-5.6
Sol through their own vendor CLIs on subscriptions, Grok 4.6 through the xAI coding CLI, and
DeepSeek V4-Pro through the vendor's first-party API, since it has no agentic CLI to run in.

The pooled ranking is Opus 5 at 0.743 pair F1, Sol at 0.660, Grok 4.6 at 0.553 and DeepSeek
V4-Pro at 0.533, identical in both realizations. That ordering is the least interesting output of
the run, and with two realizations and three tasks it carries no confidence interval.

**Citation validity is the column that separates them.** Measured as the fraction of evidence
records whose quoted excerpt appears verbatim in the document it cites, Opus 5 scored 0.991,
Grok 0.937, Sol 0.915 and DeepSeek V4-Pro 0.804. The extreme case is one cell: DeepSeek found
**every** correct entity, entity recall 1.000, and three quarters of its excerpts were not in the
documents it attributed them to, citation validity 0.250. It knew the answer and misquoted the
source, and a summary score cannot express that, because the two components moved in opposite
directions inside the same cell.

**Reproducibility is the second column, and it inverts the obvious read.** Grok 4.6 has by far the
widest spread of scores across tasks, which looks like an unstable model until the same tasks run
twice: its mean movement between two identical realizations is **0.018**, the most reproducible
arm in the set. The spread is across tasks, not between runs. DeepSeek V4-Pro moves **0.299 on
average and 0.474 at worst**, turning 0.242 into 0.667 on one task and 0.632 into 0.158 on
another, from identical inputs. A single measurement of that arm means very little.

One cell deserves its own reading, because it is the metric failing rather than the model. Grok
scored 0.053 on a task where it had returned all nineteen correct items with every excerpt
verbatim; it lost the points for abbreviating the entity names, writing the short form where the
answer key held the full sentence. The task text explicitly permitted a short label. So that cell
measures naming convention and reports it as retrieval, which is a defect in the key. The same
behaviour is still a real signal for anything that parses the output downstream, and Grok
abbreviated hardest of the four on identical instructions. Both readings stand; neither cancels
the other.

Method, both realizations cell by cell, the transports and resolved model ids, tool economy, cost
and the full caveat list are in the companion
[methodology and raw data](/investigations/four-model-retrieval-bench-method). The three tasks
were inherited from an earlier pre-registration that chose them because one model beat another
hardest on them, so this is a screening instrument rather than a neutral sample, and none of it is
an adoption verdict for anything.

What the run costs a reader who takes only the ranking is the two findings that would decide a
deployment. Both live in columns a leaderboard does not print, and both were invisible until the
same tasks ran a second time.