{"author": "Ashita Orbis", "card": "Five research questions at five effort settings on GPT-5.6 Sol and Claude Opus 5: searches rose with the setting, GPT-5.6 Sol's means varied little, and Claude Opus 5 scored lowest at extra high and highest at max.", "category": "lab-notes", "conversationExcerpts": false, "date": "2026-10-03", "description": "55 graded research outputs from GPT-5.6 Sol and Claude Opus 5 at five explicit effort settings plus the Opus default, August 2026: higher effort raised search volume, no GPT-5.6 Sol setting above medium improved its mean grade, and Claude Opus 5 averaged lowest at extra high and highest at max.", "meansEndsRatio": 0.3, "projects": [], "slug": "more-effort-more-searches", "subtitle": "GPT-5.6 Sol and Claude Opus 5 answered five research questions at five explicit reasoning effort settings; search volume climbed with the setting, mean grades moved by a few tenths, and one stale price appeared at exactly GPT-5.6 Sol's three highest settings, while its low and medium runs did not list that price.", "tags": ["lab-notes", "reasoning effort", "web search", "research agents", "benchmarking", "gpt-5.6-sol", "claude-opus-5"], "title": "More Effort Bought More Searches"}
---
GPT-5.6 Sol and Claude Opus 5 were each given five web research questions at five explicit reasoning
effort settings, to see what a higher setting changes when an agent searches. GPT-5.6 Sol ran through
OpenAI's Codex command line tool with its built in web search, and Claude Opus 5 through Claude Code
with web search and page fetching. Each ran once per question on 6 August 2026 at low, medium, high,
extra high and max, plus Claude Opus 5's unspecified default. Higher settings scaled searching while mean grades
moved by a few tenths, with GPT-5.6 Sol peaking at medium and Claude Opus 5 lowest at extra high and
highest at max.

The questions came from an earlier Exa comparison: four freshness questions that had separated its
configurations, about how AI hardware serves large models, a then scheduled API tariff change, the
cheapest long context model APIs and the newest flagship model identifiers, and an archival
question about wartime economic statistics. Claude Fable 5 graded all 55 outputs from complete reads with the earlier
comparison's five part rubric, checking at least two claims per output against primary sources. The
API does not echo Claude Opus 5's effort level back, so the evidence that it took effect is
behavioural: on the archival question it searched 12 times at low and 57 times at max.

| Effort | GPT-5.6 Sol | Claude Opus 5 |
|---|---:|---:|
| Low | 4.76 | 4.72 |
| Medium | 4.92 | 4.80 |
| High | 4.84 | 4.80 |
| Extra high | 4.84 | 4.56 |
| Max | 4.84 | 5.00 |
| Default (unspecified) | not run | 4.88 |

Table: Mean grade over five questions, each graded 0 to 5 from a single run.

GPT-5.6 Sol's search volume climbed with the setting, from 14 searches at medium to 97 at max on the
archival question and from 10 to 27 on the hardware question. The max run spent 2,564 seconds, about
43 minutes, on a question the medium run answered to the same grade in 265 seconds, and its 97
searches missed a database the medium run had found. The grading model's report qualifies this:
those searches also produced the broadest map of wartime statistical serials of any GPT-5.6 Sol
run, despite the equal grade.

A pricing error repeated across GPT-5.6 Sol runs, and the same report calls its adjudication of that
price the emblematic finding of the whole program. At high, extra high and max, GPT-5.6 Sol priced
GPT-5.6 Luna at $1.00 per million input tokens and $0.10 cached while linking
the OpenAI page that listed $0.20, $0.02 and $1.20. The $1.00 figure was the launch price, and OpenAI's
changelog records an 80% cut to GPT-5.6 Luna starting 30 July, seven days before the run. The Claude
Opus 5 runs that fetched the live page reported the current price. Six Exa configurations in the
earlier comparison had agreed on about $0.40. The grading model's report suggests that this figure was
most plausibly the long context rate for prompts over 272,000 input tokens repeated without its
condition, and concludes that, in this pricing check, agreement among search results was weaker
evidence than one fetch of the vendor's page.

The grading model's report says that what separated the Claude Opus 5 settings was how their outputs
handled what they could not verify. At extra high they made refuted absence claims on two questions
and carried stale prices on a third. One of those
absence claims was that Google's Ironwood TPU had no public price entry when the pricing page listed
$12.00 per chip hour on demand in Iowa. At max no spot check refuted anything asserted, and its 5.00 was the only perfect
configuration mean in either comparison. At low, GPT-5.6 Sol assumed a standard cache discount and
priced a DeepSeek-V4-Flash cache hit at $0.014 per million input tokens, where the run's own check of
DeepSeek's pricing page recorded $0.0028.

The rubric scores accuracy at zero when a checked claim is materially wrong, but the grades gave such
outputs partial credit, usually three points, as the Exa comparison had. Scoring every output with a
refuted spot check at zero accuracy leaves GPT-5.6 Sol at 4.92 on medium and 4.64 to 4.72 elsewhere,
and puts Claude Opus 5 between 4.16 at extra high and 5.00 at max, so the peaks and the trough hold.

Each cell is one run, the grader comes from the same model family as Claude Opus 5, and an instruction
file already in Claude Opus 5's context overlapped the flagship model question, so a tenth or two
between settings is not a ranking. The grading model's report recommends GPT-5.6 Sol at medium and
Claude Opus 5 at max where a wrong assertion is expensive, and it advises that when a research agent
at a high effort setting claims something does not exist, the claim stays unverified until a second
search tool or a direct fetch of the source confirms it. The per question grades, the strict
scoring and the price adjudication are in the companion
[method, per question grades and pricing adjudication](/investigations/search-agent-effort-ladders-method).