How the Claude Code CLI Harness Influences Model Behavior and Drives Cost

Ask an engineering team what their AI coding assistant costs to run and you will get last month’s invoice. If you ask them why the number moved, the answers will probably be vague: the model got better, usage went up, someone left an agent running. I wanted a real answer, so I built a model behavioral analysis program with Model Analytics capabilities, including a benchmark that runs itself every night against Claude Code, Anthropic’s coding agent. I have been collecting the results for months. The finding that surprised me: the harness around the model moves my cost far more than the model itself does.

The setup: nine tasks, frozen forever

The benchmark is deliberately boring. It is a catalog of nine small coding tasks, each isolating one thing an agent actually does: read a file, search a codebase, fix an off by one bug, rename a function across call sites, add a feature flag, fix a failing test, plan a refactor without touching code, and run a full edit, test, and verify loop. One task is nothing but arithmetic on three tokens of output, which exists to measure the fixed overhead floor of a single invocation.

The catalog has one iron rule: once a task is committed, it is never edited. If a prompt changes even slightly, every comparison against past runs is invalid. A revision means a new task id, never an in-place edit to an old one. That rule is what makes a May result comparable to a July result.

How it works: a watcher, a clean room, and repeat runs

Each night, a watcher checks whether a new version of the Claude Code CLI has shipped. If it has, the harness builds a clean room Docker image pinned to that exact version and runs the full suite against it, headless, no human in the loop. Each task is run multiple times per version, not once, so every measurement carries a run-to-run deviation alongside its value; a result that will not reproduce across its own repeat runs never gets compared across versions. Every run records tokens in and out, cache behavior, cost, wall clock time, tool calls, and how the agent behaved along the way. The suite has now covered more than twenty CLI releases and six Claude models across over eight hundred recorded sessions.

A short detour: the four kinds of tokens

To read the findings you only need one piece of billing mechanics. Claude meters four token types, at very different prices:

• Input tokens are everything sent to the model: your prompt, the code it is shown, the agent’s instructions. Billed at the full input price.
• Output tokens are everything the model writes back. The most expensive type.
• Cache-write tokens are input the system stores for reuse, billed at a small premium over normal input.
• Cache-read tokens are stored input replayed from cache on later calls, billed at a small fraction of the full input price.

An agent like Claude Code resends a large, mostly identical context on every API call. Whether that repeated context is metered at the full input price or the cache-read price is almost the entire cost story that follows.

What the data showed

Running the nine task suite on Claude Opus 4.8 cost 2.91 dollars at CLI version 2.1.201. Fourteen releases later, at version 2.1.215, the identical suite on the identical model cost 40 cents. Same tasks, same model, 86 percent cheaper.

The mechanism is the caching cliff, and it is worth staring at. Through version 2.1.206, an average session sent roughly 3,000 input tokens at full price. From 2.1.207 onward, that number drops to single digits: 11 tokens, then 6, then 10. Not a discount, not a trim; a category of spend, effectively erased overnight by a prompt-caching improvement in the CLI. The model did not change at all.

The harness moves more than cost. The behavioral measures shift between releases too: the same task completes in fewer tool calls on newer CLI versions, meaning the harness changed how the model works, not just what the work costs. When your agent gets a version bump, you are not running the same system with a cheaper meter; you are running a different system.

The reverse also happens. Around version 2.1.196 the suite cost on one model spiked past eight dollars and stayed elevated for several releases before falling back near one dollar. If you only watched your invoice, you would have blamed the model or your team’s usage. The benchmark shows it was the tooling, and that it was later fixed.

The part I now consider mandatory: checking the ruler

A benchmark that runs for months has a credibility problem. If the same task on the same pinned version costs a different amount today than it did in May, is the model drifting under me, or is my measurement noisy? The suite answers this by rerunning old pinned images as a control. A version rerun today lands within about ten percent of its original recording, and the big version to version effects reproduce almost exactly, a 33 percent drop today versus 34 percent in the original data. That control is what lets me say the CLI is the cost driver with a straight face. It also caught a real scare: an early control run suggested the model itself had changed, and a second run showed the first was noise. The benchmark disproved its own false alarm, which is exactly the behavior you want from your measurement system before you trust it.

Why this matters beyond my setup

If your team runs AI coding tools at any scale, your cost story is probably a tooling story wearing a model costume. Version to version changes in the agent harness, caching behavior, and prompt assembly can move your unit economics by multiples while the model underneath stays identical. Three practical takeaways from my data: pin and record your tool versions the way you pin dependencies, because the version is a first class cost variable. Measure unit cost on a fixed workload, not spend on a moving one. And rerun an old configuration occasionally to prove your ruler still measures what it measured in May.

The live numbers behind this article, including the full cost trend across CLI versions and per model activity, are on the Model Analytics dashboard, which updates as new benchmark cycles land.