Claude Code CLI 2.1.271 Analysis

Measured 2026-09-15 00:46 UTC from this cycle’s benchmark runs, written up from those measurements alone, then checked by an independent reviewer. Where the wording and the tables disagree, the tables are correct.

Verdict

At the high reasoning setting, one benchmark task costs, cheapest to most expensive:

• haiku-4-5 $0.0138 per session
• sonnet-5 $0.0293 per session
• opus-4-8 $0.0469 per session
• opus-5 $0.0733 per session
• fable-5 $0.1044 per session
• fable-5-1 $0.1098 per session

At the medium setting:

• sonnet-5 $0.0273 per session
• opus-4-8 $0.0469 per session
• opus-5 $0.0604 per session
• fable-5-1 $0.0942 per session
• fable-5 $0.0987 per session

Across the last 10 cycles at the high setting (9 cycles for fable-5-1):

• opus-5 drifted up, staying inside $0.0662 to $0.0733
• fable-5 showed no clear direction, staying inside $0.1018 to $0.1139
• fable-5-1 showed no clear direction, staying inside $0.1027 to $0.1098
• haiku-4-5 showed no clear direction, staying inside $0.0138 to $0.0148
• opus-4-8 showed no clear direction, staying inside $0.0457 to $0.0496
• sonnet-5 showed no clear direction, staying inside $0.0290 to $0.0314

Across the last 10 cycles at the medium setting (9 cycles for fable-5-1):

• fable-5 drifted up, staying inside $0.0917 to $0.0987
• opus-4-8 drifted up, staying inside $0.0435 to $0.0469
• opus-5 drifted up, staying inside $0.0502 to $0.0604
• fable-5-1 showed no clear direction, staying inside $0.0926 to $0.0960
• sonnet-5 showed no clear direction, staying inside $0.0265 to $0.0282

This cycle moved from CLI version 2.1.270 to 2.1.271, and the families with a like-for-like comparison moved as follows:

• haiku fell 4.0% at the high setting
• sonnet fell 1.4% at the high setting
• sonnet rose 2.3% at the medium setting

In all three cases the individual runs behind those averages overlap heavily, so the moves do not stand out from normal run-to-run variation.

One more thing worth knowing: the spread between families is far larger than any of this cycle’s moves. At the high setting, haiku-4-5 costs 87.5% less per task than fable-5-1, and that gap is large enough to stand clear of run-to-run variation.

Which cost figure is which – setting high

The same sessions summarized three ways. They disagree on purpose, and none is the corrected version of another.

• Mean $/session: multiplies out to real spend, which makes it the budgeting number, and it is what the Cost Trends chart plots.
• Median $/session: shrugs off an unusually cheap or expensive run, but it does not sum to real spend.
• Suite cost: the price of one full pass over every prompt in the catalog, the per-prompt medians added up, so it answers what a whole run costs rather than what one session costs.

family	models compared	mean $/sess	mean delta %	median $/sess	median delta %	suite $	suite delta %	prompts
haiku	haiku-4-5	0.0143 -> 0.0138	-4.0%	0.0144 -> 0.0155	+7.5%	0.1207 -> 0.1170	-3.0%	9
sonnet	sonnet-5	0.0297 -> 0.0293	-1.4%	0.0327 -> 0.0319	-2.4%	0.2481 -> 0.2487	+0.3%	9

Suite cost uses only the prompts both cycles ran, so the two sides always add up the same catalog.

Which cost figure is which – setting medium

The same sessions summarized three ways. They disagree on purpose, and none is the corrected version of another.

• Mean $/session: multiplies out to real spend, which makes it the budgeting number, and it is what the Cost Trends chart plots.
• Median $/session: shrugs off an unusually cheap or expensive run, but it does not sum to real spend.
• Suite cost: the price of one full pass over every prompt in the catalog, the per-prompt medians added up, so it answers what a whole run costs rather than what one session costs.

family	models compared	mean $/sess	mean delta %	median $/sess	median delta %	suite $	suite delta %	prompts
sonnet	sonnet-5	0.0267 -> 0.0273	+2.3%	0.0289 -> 0.0313	+8.2%	0.2178 -> 0.2287	+5.0%	9

Suite cost uses only the prompts both cycles ran, so the two sides always add up the same catalog.

What these terms mean

New to this report? Here is the vocabulary used below, in plain words. Skip ahead if you already know it.

• Session (also “task”): one complete run of one prompt by one model, start to finish. Every cost in this report is per session.
• Family: a model line, such as opus or sonnet. One family can cover more than one version, for example opus-4-8 and opus-5.
• Cache, warm and cold: the tool reuses recent context instead of sending it again, which is far cheaper. A session that gets to reuse it is “warm”; one that has to send everything fresh is “cold”. Whether a given session lands warm or cold is partly luck, and that luck moves the cost.
• Cache luck (or cache-hit luck): the cost swing caused by how many sessions happened to land warm rather than cold, rather than by any real change in the tool or the model.
• Blended cost: cost measured across every session, warm and cold alike. This is the real money figure, and it is what the cost feed shows. “Blended” says which sessions are counted, not how they are averaged; see “Which cost figure is which” above for the averaging.
• Warm-only cost: cost measured across warm sessions only. Removing the cold ones removes most of the luck, so this is the fair test of whether cost actually changed.
• Pooled: all sessions for a family added together into a single number, rather than split out per prompt.
• Mean cost per session: the ordinary average across every session. It multiplies out to real spend, so it is the budgeting figure and the one this report leads with.
• Median cost per session: the middle session once they are lined up by cost. It ignores an unusually cheap or expensive run, but it does not add up to real spend.
• Suite cost: the price of one full pass over every prompt in the catalog, found by taking each prompt’s median and adding those up. It is a total for the whole catalog, not an average per session, so it is a larger number than either figure above and is not comparable to them.
• Cell: one family and one prompt combination, for example haiku on the rename-refactor prompt. Cells hold few sessions, so a single cell is noisy.
• n: how many sessions sit behind a number. A larger n is more trustworthy.
• p-value, and “significant”: the same prompt run twice costs different amounts, so a p-value asks whether two sets of runs can be told apart at all. Line up every run from both cycles: if one cycle’s runs sit consistently higher, p is near 0 and the difference is called “significant”; if the runs are mixed together, p is near 1 and the two sets look like the same thing measured twice. Precisely, it is the chance of seeing a gap at least this lopsided if the two cycles were truly identical. A high p is not weak evidence of a change, it is no evidence either way.
• API-equivalent cost: dollars calculated at published API rates. It is not necessarily dollars billed, because a subscription plan may already cover the usage.

Pooled blended comparison

These are the headline figures: the average real spend on one task, counting every session in the cycle, including the unlucky ones that started with a cold cache. Each setting is compared against its own previous run at the same setting. Only families that ran in both cycles at a given setting can be compared.

High reasoning setting, nightly-2.1.270 to nightly-2.1.271:

family	models compared	n (prev -> cur)	prev $/sess	cur $/sess	delta %	Mann-Whitney p
haiku	haiku-4-5	53 -> 53	0.0143	0.0138	-4.0%	0.726
sonnet	sonnet-5	53 -> 53	0.0297	0.0293	-1.4%	0.922

Medium reasoning setting, nightly-medium-2.1.270 to nightly-medium-2.1.271:

family	models compared	n (prev -> cur)	prev $/sess	cur $/sess	delta %	Mann-Whitney p
sonnet	sonnet-5	53 -> 54	0.0267	0.0273	+2.3%	0.616

Pooled warm comparison (significance control)

Whether a cost move is believable is a separate question from how big it looks. This second view keeps only the sessions that ran with a warm cache, which removes the random luck of a cold start, and compares the individual run costs from each cycle against each other. It is the check to trust when asking whether cost actually changed.

High reasoning setting:

family	models compared	n (prev -> cur)	prev $/sess	cur $/sess	delta %	Mann-Whitney p
haiku	haiku-4-5	45 -> 45	0.0132	0.0129	-2.7%	0.806
sonnet	sonnet-5	45 -> 45	0.0276	0.0271	-1.8%	0.904

Medium reasoning setting:

family	models compared	n (prev -> cur)	prev $/sess	cur $/sess	delta %	Mann-Whitney p
sonnet	sonnet-5	45 -> 45	0.0239	0.0250	+4.7%	0.529

Every one of these comparisons sits far above the 0.05 line. That does not mean nothing changed: it means this measurement cannot tell these moves apart from the ordinary variation between runs.

What the reasoning setting costs

The same release was measured twice, once at the high reasoning setting and once at medium. Comparing the two pooled sets of session costs for each model gives the following, with medium measured against high and a negative percentage meaning medium is the cheaper setting.

model	high mean $	medium mean $	change	p	n
fable-5	0.1044	0.0987	-5.5%	0.275	53/55
fable-5-1	0.1098	0.0942	-14.3%	0.141	53/54
opus-4-8	0.0469	0.0469	-0.1%	0.711	53/56
opus-5	0.0733	0.0604	-17.6%	0.165	56/61
sonnet-5	0.0293	0.0273	-6.8%	0.271	53/54

Medium is the cheaper setting in every row, but no row clears the 0.05 line:

• opus-5 is 17.6% cheaper at medium, and the difference cannot be told apart from run-to-run variation
• fable-5-1 is 14.3% cheaper at medium, and the difference cannot be told apart from run-to-run variation
• sonnet-5 is 6.8% cheaper at medium, and the difference cannot be told apart from run-to-run variation
• fable-5 is 5.5% cheaper at medium, and the difference cannot be told apart from run-to-run variation
• opus-4-8 is 0.1% cheaper at medium, and the difference cannot be told apart from run-to-run variation

haiku-4-5 was measured at the high setting only, so it has no cross-setting figure.

Notable cells (secondary)

At the high setting, no individual prompt moved beyond the noise threshold. At the medium setting, two sonnet-5 prompts moved more than the rest:

• fix-off-by-one, which fixes a one-line off-by-one bug in a small Python function, rose 30.8% on warm runs
• that prompt ran 6 times in each cycle, at p=0.251
• rename-refactor, which renames a function everywhere it appears, in its definition, its export, and every call site across several files, rose 22.4% on warm runs
• that prompt ran 6 times in each cycle, at p=0.465

family - prompt	warm cost delta %	n (prev/cur)	cost p	blended delta %
sonnet - fix-off-by-one	+30.8%	6/6	0.251	+13.8%
sonnet - rename-refactor	+22.4%	6/6	0.465	+25.1%

Both sit above the 0.05 line. They also come from nine prompt comparisons run at once. With that many tests, roughly one apparent hit per twenty cells is expected by chance alone. Treat them as something to watch next cycle, not as a result.

Caveats

• Two cost views are reported. The headline is the real per-task spend across every session, including cold-start cache misses, and it is what the dashboard shows. The warm-only view strips out that random cache luck and is the control for whether a change is real. A headline move with no matching warm-only signal may simply be cache luck rather than a genuine change.
• The two cycles being compared ran different CLI versions (2.1.270 then 2.1.271), which is the point of the benchmark: any pooled difference measures the effect of the CLI version. It should not be read as a statement about the models themselves without a separate test that holds the CLI version fixed.
• Individual prompt results come from nine simultaneous comparisons. With that many tests, roughly one apparent hit per twenty cells is expected purely by chance, so a single standout cell is not a finding on its own.
• Token capture for Haiku is known to be incomplete, so its cost and token figures should be read as lower bounds.
• Timings are wall-clock, including the time the model spends thinking, and costs are calculated as API-equivalent amounts rather than dollars actually billed.

---

Verification appendix

• Verifier verdict: PASS after 3 round(s).
• A first draft was rejected and revised; the objections were resolved.