A working note · 14 September 2026
What a harness is, and why it decides your bill
Every benchmark number published this month is a score for a model plus a harness. Almost nobody separates the two, and the half nobody talks about is the half you control.
01What actually runs when you "use a model"
A model on its own does nothing. What ships is a pipeline, and the model is one stage in it.
Stages one to three and five to six are chosen by you. Stage four, the model, is chosen by the vendor.
| Stage | What it is | Chosen by |
|---|---|---|
| [1] | The job, and how it gets broken into steps | You |
| [2] | System prompt, examples, how much context you carry between turns, whether you compact or truncate | You |
| [3] | Which tools it can call, how they are described, what they return when they fail | You |
| [4] | The model itself | The vendor |
| [5] | How many attempts, what happens after a bad tool call, whether it can see its own error and self-correct | You |
| [6] | What counts as success | You |
The harness is stages 1, 2, 3, 5 and 6. The model is stage 4. When people argue about Astra against Fable, they are arguing about one stage out of six, and it is the only one they cannot change.
The analogy
The model is the engine. The harness is the chassis, the tyres, the pit crew and the race strategy. Two teams can bolt in the identical engine and finish a lap apart. Nobody watching would conclude the engine was the difference. A benchmark headline concludes exactly that.
02The effect is not small
Same model. Same benchmark. Same week. Only the harness changed.
Reported by ARC Prize, which states the two conditions must be labelled separately. The accent segment is the portion of the score attributable to scaffolding rather than to the model.
What changed between those two bars: the provider adapter keeps the model's reasoning state between requests instead of discarding it and re-deriving it each turn, and compacts context rather than truncating it. No retraining. No different model. It simply stops making it start from scratch.
The honest range. Fable moves 5.9 points between those two harnesses. Astra moves 0.91. So the effect is real and sometimes enormous, but it is not uniformly 37 points, and the direction of the vendor gap reverses depending on whose harness you use.
03Why this is the part that saves money
| The point | |
|---|---|
| 1 | Migration is an expensive answer to the wrong question. Switching vendor means re-testing every prompt, rewriting tool definitions, learning new failure modes and new rate limits. Weeks of work. The gap being chased is often two points. The gap available from your own scaffold is often twenty. |
| 2 | Your invoice is a harness output, not a rate-card output. On the official Terminal-Bench leaderboard the same benchmark cost $3,267 to complete with one setup and $6,244 with another. If you are not caching, not compacting and retrying blindly, you are paying a multiple for identical work. |
| 3 | It explains why the benchmarks contradict each other. Once you know the harness is inside the number, "Astra 59, Fable 52" and "Astra 58.2, Fable 57.9" stop being a contradiction. They are two different cars carrying the same two engines. This is how you stay calm when somebody quotes a table at you. |
The same lesson, from the billing side
Anthropic's headline advantage on Fable 5.1 is a cache read priced at $0.25 per million against OpenAI's $1.00, four times cheaper. For the first fortnight after launch a bug meant context attached after tool results was not being cached at all, so it was re-sent as full-price input on every turn. The rate card said four times cheaper. The harness meant it was not. A price list is not an invoice.
04The line worth saying out loud
Every benchmark number you have seen this month is a score for a model plus a harness. Nobody has published a clean model-only comparison, because there isn't one. So before you migrate, go and measure your own harness. That is the variable you actually control.
What to do on Monday
Pick one workload you already run. Measure three things before you change any vendor: what fraction of your input tokens are cache hits, how many retries a typical task takes, and how much context you are re-sending each turn that has not changed. If any of those look bad, fix them first. That work is cheaper than a migration, it is reversible, and it carries over whichever model you end up on.
The caveat, before somebody else supplies it
This cuts against my own argument too. The 37-point swing is one benchmark, and ARC Prize explicitly says those two conditions are not comparable, which is precisely why they must be reported separately. The 0.9 to 5.9 point range on Terminal-Bench is the more typical magnitude. The claim that survives is not "the harness is always worth 37 points". It is: the harness is inside every number you read, it is frequently larger than the vendor gap being argued about, and it is the only part of the pipeline you own.