{"author": "Ashita Orbis", "category": "lab-notes", "conversationExcerpts": false, "date": "2026-08-11", "description": "The opus CLI alias resolved to the previous model on the day Opus 5 shipped, so a benchmark that trusts the alias silently tests the wrong thing. Re-derived from raw transcripts: 8,121 assistant messages, 100% the intended model, and one arm nobody checked.", "draft": false, "meansEndsRatio": 0.25, "projects": [], "slug": "which-model-actually-ran", "subtitle": "A benchmark's Opus arm was challenged as possibly being the previous model, and 8,121 assistant messages settled it; the check that settled it is the one the other arm never got.", "tags": ["lab-notes", "benchmarking", "model identity", "claude-code", "measurement"], "title": "Which Model Actually Ran"}
---
A model id passed at launch is a request rather than a record, because the short names most
tooling accepts resolve through configuration that the experiment neither owns nor archives. On
the machine that ran a July benchmark of Claude Opus 5 against GPT-5.6 Sol, the `opus` alias in
Claude Code resolved to Claude Opus 4.8 throughout the day the newer model became available. An
arm launched under that alias would have measured the previous generation while every directory
name and summary line in the results said otherwise.

The benchmark was challenged on exactly that ground once it reported a ranking that reversed an
earlier one. Reading the launch scripts back would have been circular, since a launch script is
the artifact under suspicion, so the question went to the session transcripts instead, where
every assistant message carries its own model field. Across the 1,105 Opus arm cell directories
of the first two sub-experiments, 8,121 assistant messages name Opus 5 and no message names
anything else. In the third, 70 of the 87 admitted cells retain durable transcripts totalling 401
messages with no drift, and the other 17 rest on a ledger the scoring gate wrote before a reboot
cleared the store.

That gate is the part worth copying. The design frozen before the run states the alias hazard in
writing, and the rule built on it admits a cell only when a transcript exists and every message
in it names the intended model. A rule of that shape can reject its own experiment, and this one
did, marking 141 cells unverified and excluding them from every reported number. The check costs
one field lookup per message. It is worth building before the results exist, because a benchmark
that trusts the alias produces numbers no later analysis can repair, the identity of the system
under test having never been written down in the outputs at all.

The rule was also written for one arm only. All 78 cells of the Sol arm carry the literal string
`n/a` in the gate's model column, admitted on the strength of their launch line, which inverts
the provenance claim the benchmark's own summary made. The arm with the weaker paperwork turned
out to be the arm nobody had checked. Sol identity does hold up on external evidence, since the
Codex rollout logs for that window name one model and no other, but the instrument did not
establish it and an outside pass did.

The earliest message anywhere on that machine naming Opus 5 is timestamped 2026-07-24T17:46:55Z,
and the benchmark's first Opus cell transcript begins eighteen minutes later. Identity is
therefore a dating problem as much as a configuration one: any artifact on that box older than
that timestamp which says "opus" means 4.8 regardless of its label, and the same arithmetic runs
on any machine where an alias quietly changed meaning. Method, the per-arm adjudication table,
the reproduction commands and the limitations are in the companion
[methodology and raw data](/investigations/model-identity-verification-method).

One number from that benchmark is deliberately absent here. The comparative result those cells
feed is confounded by a reasoning effort mismatch between the arms and its clean re-run is still
outstanding, so the identity finding travels alone, being the only part of that experiment
currently settled.