{"author": "Ashita Orbis", "category": "lab-notes", "conversationExcerpts": false, "date": "2026-08-11", "description": "Thirteen cells, 468 runs, 43 minutes and $3.03: the Hermes optimisation batch reproduces at close to published magnitudes on weak models, attenuates rather than vanishes on a strong one, does nothing for DeepSeek, and carries a runaway-generation tail.", "draft": false, "meansEndsRatio": 0.35, "projects": [], "slug": "what-replicated", "subtitle": "An independent 468-run replication of Nous Research's Hermes efficiency claim landed close to the published magnitudes on the weak models, contradicted the parity claim on the strong one, and found a counter-example on the cheap workhorse.", "tags": ["lab-notes", "agent harness", "replication", "benchmarking", "measurement"], "title": "What Replicated, and What Did Not"}
---
Teknium of Nous Research [announced on 2026-08-02](https://x.com/Teknium/status/2084065915004747888)
that Hermes Agent had become dramatically more efficient, especially for smaller and weaker
models, attaching an A/B card that reports about 21% fewer agent turns, 29% fewer tool calls and
23% less wall clock on a weak model, with parity on a strong one. A claim of that shape is cheap to check
independently and rarely is, so it was checked: thirteen cells, twelve fresh tasks in three
families, three repetitions each, **468 runs in 43 minutes for $3.03**, with the harness version
the only thing varying inside the paired columns.

**It reproduces on the weak models, at close to the published magnitudes, on a task set the
batch has never seen.** Holding model and endpoint fixed, the newer harness against the older
one gives **27.6% fewer agent turns, 30.2% fewer tool calls, 44.3% fewer tool errors and 35.0%
lower median wall clock on Gemini 3.5 Flash-Lite**, and **21.1% fewer turns on GPT-5.6 Luna**.
Against a published card of 21%, 29% and 23%, that is an independent replication landing inside
a few points on a different operator's instrument.

**The half that did not reproduce is the parity claim.** On GPT-5.6 Sol the same upgrade still
buys 19.0% fewer turns and 9.0% lower median wall clock, which is attenuation rather than
absence. The weak and strong split the card describes shows up here as a gradient rather than a
cliff, large on the two weak models, moderate on the strong one, and the honest reading is a
partial non-replication. The efficiency half lands; the parity half does not.

The counter-example is sharper than either. On DeepSeek V4 Flash the upgrade is near parity on
every behavioural axis, at 2.4% fewer turns, and it costs **21% more tokens per solved task**.
Weak models benefit is too coarse a rule, since one of the two cheap workhorses tested does not
benefit at all.

One behaviour should drive any upgrade decision more than the averages do. Three of 36 runs on
the newer harness emitted **exactly 65,536 output tokens in a single API call**, taking between
341 and 643 seconds, against zero such runs in 36 on the older one. Every other call in those
same runs ran between 61 and 284 tokens. All three passed, so no correctness metric can see it,
and it turns that cell's mean wall clock into a 127% regression while its median reads as a
16.6% improvement. Those three calls are 51% of the cell's entire spend.

Because every cell routed through one counting proxy to one upstream, the cross harness
comparison is exact rather than approximate, and harness choice moves **cost per solved task by
about 2.7 times on an identical model with identical serving**. On both cheap models the
cheapest harness per solved task was Hermes.

No quality claim survives this run. **467 of 468 runs passed**, a ceiling effect the design
pre-registered as a risk, so the tasks cannot separate these harnesses on correctness and the
pass rate column is not a capability ranking. Method, all thirteen cells, the paired version
tables, the task families and the reproduction rig are in the companion
[methodology and raw data](/investigations/harness-replication-method).

What the run taught about instruments outweighs what it taught about harnesses. Four measurement
artefacts were caught before publication, three of them instrumentation masquerading as harness
behaviour: a 58% latency regression that was the proxy retrying a probe it had failed to serve, a
multi-minute startup penalty that was one harness paying a database migration on every fresh home
directory, and a beautifully clean deterministic failure fingerprint that turned out to be the
grader matching on a transcript the instrument itself had written into the graded directory. All
three flattered a story the operator already believed, and that is the direction measurement
error takes when nobody is looking for it.