{"author": "Ashita Orbis", "category": "lab-notes", "conversationExcerpts": false, "date": "2026-08-11", "description": "GPT-5.6 Luna and GPT-5.3 Spark at effort low, given a plain-language prompt, against hand-written keyword and regex classifiers on four real classification sites, independently adjudicated, with a split verdict and a batching correction.", "draft": false, "meansEndsRatio": 0.35, "projects": [], "slug": "a-cheap-model-against-a-regex", "subtitle": "Four classification tasks, 118 items, production code paths as the baseline: the cheap model wins two by thirty and forty points, loses one, and the site that matters most keeps its regex anyway.", "tags": ["lab-notes", "classification", "cheap models", "benchmarking", "measurement"], "title": "A Cheap Model Against a Regex"}
---
Most working automation classifies natural language with a word list, because a word list is
deterministic and free, and the question of whether a cheap model would do it better is usually
argued rather than measured. Four such classifiers were measured against two cheap models at
their lowest reasoning effort, given nothing but a plain description of the task and an
instruction to reply with one line of JSON.

The baselines are the production code paths themselves, imported rather than reimplemented, so
the keyword column is what the running system actually decides. The items are 118 pieces of real
text sampled deterministically from live corpora, hand labelled before any model ran. A stronger
model then adjudicated every one of the 53 items where any system disagreed with the hand label,
audited ten agreed items for silent errors, and recomputed one task's arithmetic; it flipped
eight hand labels, several of them traceable to labelling from a truncated view, and excluded one
malformed item. The scores below use its adjudicated labels rather than mine.

| Task | n | keyword | Spark at low | Luna at low |
|---|---|---|---|---|
| Does this log row claim the work is finished | 29 | 62.1% | 89.7% | **93.1%** |
| Which state is this stalled row in, of three | 29 | 51.7% | 89.7% | **93.1%** |
| Does this free-text answer contain a new instruction | 38 | **84.2%** | 78.9% | 81.6% |
| Does this status file claim readiness | 20 | 75.0% | **90.0%** | **90.0%** |

Both models returned clean single-line JSON on all 236 calls, with no parse failures.

The error asymmetry matters more than the totals. Of the twelve rows the keyword tool escalated
for a human decision, **ten were semantic non-claims**: phrases reporting that some step had
landed, which the word list read as claims that the whole item was done. The cheap model made
that error once. After the hybrid shipped, the escalation pile on that surface fell from 110
items to 47, and a second classifier that had found zero instances of one category in 263 rows
found 24.

Cost is what makes it deployable rather than interesting. A single batched call carrying all 29
rows of one task used about 24 times fewer tokens than 29 separate calls, and moved the weekly
subscription window by less than a whole percentage point. At this volume that is free.

Batching does cost accuracy, and the honest number is the shipped one:
**putting both judgments in one batched call cost about seven points on the first task, taking
93.1% to 86.2% in production**, against a keyword baseline that did not move.

Two of the sites keep their keyword lists, and the reasons are not about accuracy. One detects
whether a model provider is rate limiting the caller, where a model call to detect that the
models are unavailable is circular, so the word list stays as the floor. The other decides
whether staged content is safe to publish, where the failure mode is a leaked credential, and a
model must never be the only gate on a publish path. **Keywords as the non-negotiable floor
wherever a miss means a leak or a false green, semantic as an additional sweep, never a
replacement.** Method, the site inventory, the adjudication protocol, the cost measurement and
the per-site verdicts are in the companion
[methodology and raw data](/investigations/semantic-vs-keyword-method).

The task the keyword pass won deserves the same discipline as the ones it lost. Its 84.2% against
81.6% is well inside the noise of a 38 item single run, so the correct reading is that no semantic
advantage was demonstrated there, rather than that keywords were proven better.