Claude Benchmarks
Suite cost (USD)
- sonnet-5 (high)
- fable-5 (high)
- haiku-4-5
- opus-4-8 (high)
- opus-5 (high)
- fable-5-1 (high)
- opus-5-5 (high)
- sonnet-5-5 (high)
- haiku-5-5
- sonnet-4-6 (high)
- fable-5-2 (high)
Data table
| Claude Code CLI version | sonnet-5 (high) | fable-5 (high) | haiku-4-5 | opus-4-8 (high) | sonnet-5 (medium) | opus-5 (high) | opus-5 (low) | opus-5 (max) | opus-5 (medium) | opus-5 (xhigh) | fable-5 (medium) | opus-4-8 (medium) | fable-5 (max) | fable-5 (xhigh) | opus-4-8 (max) | opus-4-8 (xhigh) | sonnet-5 (max) | sonnet-5 (xhigh) | fable-5-1 (high) | fable-5-1 (medium) | fable-5-1 (max) | fable-5-1 (xhigh) | opus-5-5 (high) | opus-5-5 (medium) | sonnet-5-5 (high) | sonnet-5-5 (medium) | haiku-5-5 | sonnet-4-6 (high) | sonnet-4-6 (medium) | fable-5-2 (high) | fable-5-2 (medium) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2.1.196 | 0.39959 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.202 | 0.307441 | 1.585473 | 0.138659 | 0.787706 | 0.292861 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.204 | 0.43591 | 1.589345 | 0.136938 | 0.7473 | 0.423905 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.205 | 0.401936 | 1.412947 | 0.137007 | 0.68471 | 0.399699 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.206 | 0.415858 | 1.452328 | 0.134095 | 0.711829 | 0.411794 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.207 | 0.313161 | 1.169142 | 0.151735 | 0.494431 | 0.292131 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.208 | 0.301529 | 1.052804 | 0.137217 | 0.473321 | 0.271329 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.210 | 0.282338 | 1.052505 | 0.135879 | 0.478824 | 0.261952 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.211 | 0.304312 | 1.042501 | 0.135169 | 0.441384 | 0.257755 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.212 | 0.304967 | 1.037562 | 0.131874 | 0.449024 | 0.266073 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.214 | 0.298263 | 1.00327 | 0.140986 | 0.470604 | 0.270236 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.215 | 0.289674 | 0.98978 | 0.129231 | 0.403363 | 0.265379 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.216 | 0.269861 | 0.932445 | 0.143514 | 0.419973 | 0.250677 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.217 | 0.29384 | 0.970618 | 0.138043 | 0.405417 | 0.256946 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.218 | 0.284298 | 0.993526 | 0.128458 | 0.402994 | 0.265344 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.219 | 0.286448 | 1.040209 | 0.132845 | 0.408153 | 0.261634 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.220 | 0.276864 | 0.987417 | 0.125554 | 0.40121 | 0.264421 | 0.657846 | 0.432675 | 0.815055 | 0.566123 | 0.762884 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.221 | 0.286049 | 0.977372 | 0.131999 | 0.401626 | — | 0.65512 | — | 0.873209 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.222 | 0.29028 | 1.098188 | 0.131929 | 0.443637 | 0.268545 | 0.646637 | — | — | 0.56267 | — | 0.892297 | 0.380577 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.223 | 0.296099 | 0.997915 | 0.136063 | 0.419563 | 0.26388 | 0.654669 | — | — | 0.563948 | — | 1.019459 | 0.417344 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.224 | 0.292122 | 1.085281 | 0.131863 | 0.441856 | 0.264969 | 0.662316 | — | — | 0.574012 | — | 0.931341 | 0.397382 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.225 | 0.29478 | 1.084727 | 0.136727 | 0.424706 | 0.26217 | 0.68277 | — | — | 0.582747 | — | 0.945328 | 0.392409 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.226 | 0.298991 | 1.059551 | 0.143038 | 0.435342 | 0.262172 | 0.688851 | — | — | 0.593155 | — | 0.924563 | 0.423615 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.227 | 0.296432 | 1.074569 | 0.132082 | 0.474679 | 0.260069 | 0.72558 | — | — | 0.567964 | — | 0.992387 | 0.43579 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.228 | 0.309403 | 1.06973 | 0.133659 | 0.422149 | 0.253376 | 0.679035 | — | — | 0.573721 | — | 0.912215 | 0.413112 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.229 | 0.30224 | 1.069366 | 0.136153 | 0.452609 | 0.259779 | 0.683558 | — | — | 0.564781 | — | 0.909137 | 0.424274 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.231 | 0.301637 | 1.046028 | 0.134929 | 0.442898 | 0.261313 | 0.709022 | — | — | 0.574353 | — | 0.946331 | 0.407458 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.232 | 0.296741 | 1.046169 | 0.126988 | 0.448157 | 0.263367 | 0.636067 | — | — | 0.593776 | — | 0.934624 | 0.429926 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.233 | 0.299494 | 1.039348 | 0.131054 | 0.397076 | 0.263568 | 0.660987 | — | — | 0.608725 | — | 0.893037 | 0.423469 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.234 | 0.283871 | 1.049789 | 0.123664 | 0.415698 | 0.267799 | 0.627984 | — | — | 0.55549 | — | 0.884193 | 0.415903 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.235 | 0.28956 | 1.027151 | 0.128207 | 0.450877 | 0.251189 | 0.621802 | — | — | 0.533325 | — | 0.882008 | 0.436997 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.236 | 0.287515 | 0.957125 | 0.132953 | 0.454512 | 0.273248 | 0.631873 | — | — | 0.516582 | — | 0.923976 | 0.445696 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.237 | 0.295928 | 1.062191 | 0.131669 | 0.429356 | 0.254227 | 0.624628 | — | — | 0.512943 | — | 0.923061 | 0.42662 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.238 | 0.273972 | 0.959072 | 0.135126 | 0.418491 | 0.26626 | 0.620326 | — | — | 0.505544 | — | 0.902759 | 0.421965 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.239 | 0.277027 | 1.005888 | 0.132268 | 0.44007 | 0.256925 | 0.668432 | — | — | 0.521478 | — | 0.863074 | 0.412674 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.240 | 0.296605 | 1.050034 | 0.129466 | 0.464848 | — | 0.627648 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.241 | 0.284886 | 1.019615 | 0.134739 | 0.432911 | 0.264402 | 0.634396 | — | — | 0.495076 | — | 0.89114 | 0.440323 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.243 | 0.281347 | 1.065704 | 0.128777 | 0.439055 | 0.258768 | 0.639663 | — | — | 0.510795 | — | 0.895002 | 0.418189 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.245 | 0.277618 | 1.078632 | 0.127699 | 0.437461 | 0.275164 | 0.62811 | — | — | 0.506187 | — | 0.938824 | 0.4392 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.246 | 0.292935 | 1.026344 | 0.128329 | 0.430404 | 0.262555 | 0.598752 | — | — | 0.512927 | — | 0.90136 | 0.445008 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.247 | 0.274649 | 1.069361 | 0.12853 | 0.452896 | 0.262064 | 0.612011 | — | 0.881819 | 0.509641 | 0.759401 | 0.944405 | 0.431513 | 1.468829 | 2.045082 | 0.710167 | 0.549591 | 0.427245 | 0.319708 | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.248 | 0.241969 | 0.850789 | 0.116072 | 0.372271 | 0.221059 | 0.537779 | — | 0.807233 | 0.445608 | 0.674777 | 0.786564 | 0.379949 | 2.842755 | 1.903706 | 0.662301 | 0.457867 | 0.387419 | 0.280323 | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.250 | 0.243362 | 0.945936 | 0.119991 | 0.372947 | 0.22632 | 0.55282 | — | 0.828557 | 0.448003 | 0.696182 | 0.760813 | 0.354301 | 2.829469 | 1.871419 | 0.650926 | 0.45787 | 0.354036 | 0.281948 | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.251 | 0.250543 | 0.870722 | 0.118075 | 0.385571 | 0.235105 | 0.53371 | — | 0.787098 | 0.415421 | 0.72368 | 0.784472 | 0.359907 | 3.017697 | 1.857015 | 0.670861 | 0.433931 | 0.36842 | 0.282411 | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.252 | 0.25062 | 0.891468 | 0.12186 | 0.372919 | 0.236097 | 0.527568 | — | — | 0.405826 | — | 0.775454 | 0.361293 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.257 | 0.25659 | 0.87196 | 0.114944 | 0.380457 | 0.225249 | 0.525918 | — | — | 0.439004 | — | 0.738726 | 0.363476 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.258 | 0.247104 | 0.819051 | 0.11879 | 0.383242 | — | 0.535461 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.259 | 0.248202 | 0.839904 | 0.118003 | 0.386357 | 0.233196 | 0.526803 | — | — | 0.430426 | — | 0.747331 | 0.360691 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.260 | 0.248078 | 0.836241 | 0.117441 | 0.367197 | 0.223846 | 0.554445 | — | — | 0.415243 | — | 0.787331 | 0.365427 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.261 | 0.242446 | 0.867681 | 0.1213 | 0.37043 | 0.222379 | 0.554465 | — | — | 0.44107 | — | 0.74602 | 0.368507 | — | — | — | — | — | — | 0.56144 | 0.466341 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.263 | 0.246518 | 0.928523 | 0.12043 | 0.383042 | 0.229081 | 0.580929 | — | — | 0.412345 | — | 0.717638 | 0.366587 | — | — | — | — | — | — | 0.573391 | 0.454946 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.265 | 0.251823 | 0.880232 | 0.121911 | 0.386437 | 0.227746 | 0.601206 | — | — | 0.429159 | — | 0.806786 | 0.345241 | — | — | — | — | — | — | 0.542945 | 0.468057 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.266 | 0.251599 | 0.825064 | 0.118225 | 0.380074 | 0.226531 | 0.582992 | — | — | 0.439395 | — | 0.790282 | 0.358274 | — | — | — | — | — | — | 0.531119 | 0.47197 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.267 | 0.260124 | 0.913793 | 0.123804 | 0.380495 | 0.223534 | 0.584541 | — | — | 0.463294 | — | 0.756674 | 0.359009 | — | — | — | — | — | — | 0.536231 | 0.471379 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.268 | 0.247574 | 0.847196 | 0.120601 | 0.402512 | 0.233332 | 0.572696 | — | — | 0.467918 | — | 0.74891 | 0.371418 | — | — | — | — | — | — | 0.571852 | 0.479094 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.269 | 0.247532 | 0.883161 | 0.125496 | 0.37619 | 0.225884 | 0.595832 | — | — | 0.46909 | — | 0.792319 | 0.367078 | — | — | — | — | — | — | 0.549446 | 0.480273 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.270 | 0.248077 | 0.872274 | 0.120672 | 0.36345 | 0.217774 | 0.606586 | — | 0.788327 | 0.449836 | 0.69398 | 0.779055 | 0.383992 | 1.302987 | 0.970925 | 0.612335 | 0.461792 | 0.359038 | 0.280361 | 0.581184 | 0.476668 | 2.029729 | 1.025853 | — | — | — | — | — | — | — | — | — |
| 2.1.271 | 0.248705 | 0.875075 | 0.117043 | 0.379788 | 0.228741 | 0.607335 | — | — | 0.468559 | — | 0.78133 | 0.364044 | — | — | — | — | — | — | 0.602664 | 0.471878 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.272 | 0.254296 | 0.885335 | 0.12593 | 0.37832 | 0.228185 | 0.650927 | — | — | 0.472291 | — | 0.727833 | 0.367279 | — | — | — | — | — | — | 0.564856 | 0.485264 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.278 | 0.255792 | 0.876415 | 0.114951 | 0.416464 | 0.228845 | 0.63328 | — | — | 0.48095 | — | 0.735923 | 0.364167 | — | — | — | — | — | — | 0.592477 | 0.488103 | — | — | — | — | — | — | — | — | — | — | — |
| 2.1.280 | 0.252207 | 0.882213 | 0.116382 | 0.365332 | 0.230838 | 0.602693 | — | — | 0.439992 | — | 0.750666 | 0.373698 | — | — | — | — | — | — | 0.572017 | 0.47293 | — | — | 0.262347 | 0.215071 | — | — | — | — | — | — | — |
| 2.1.281 | 0.2483 | 0.866807 | 0.120914 | 0.375279 | 0.223112 | 0.622124 | — | — | 0.478544 | — | 0.7162 | 0.374442 | — | — | — | — | — | — | 0.583911 | 0.470826 | — | — | 0.270879 | 0.2318 | — | — | — | — | — | — | — |
| 2.1.282 | 0.24122 | 0.885678 | 0.11963 | 0.381661 | 0.22376 | 0.622266 | — | — | 0.483949 | — | 0.723319 | 0.361705 | — | — | — | — | — | — | 0.609478 | 0.481824 | — | — | 0.276317 | 0.225929 | — | — | — | — | — | — | — |
| 2.1.283 | 0.246044 | 0.85939 | 0.12412 | 0.392725 | 0.229795 | 0.6315 | — | — | 0.479399 | — | 0.740795 | 0.368393 | — | — | — | — | — | — | 0.583529 | 0.487228 | — | — | 0.265512 | 0.234032 | — | — | — | — | — | — | — |
| 2.1.284 | 0.247821 | 0.833854 | 0.119582 | 0.400604 | 0.2354 | 0.608328 | — | — | 0.469789 | — | 0.726728 | 0.368811 | — | — | — | — | — | — | 0.56529 | 0.486625 | — | — | 0.262267 | 0.225697 | 0.118675 | 0.101985 | — | — | — | — | — |
| 2.1.285 | 0.246052 | 0.8616 | 0.120087 | 0.381505 | 0.228927 | 0.671058 | — | — | 0.478074 | — | 0.728824 | 0.371018 | — | — | — | — | — | — | 0.572613 | 0.473703 | — | — | 0.259544 | 0.231185 | 0.11693 | 0.102549 | — | — | — | — | — |
| 2.1.286 | 0.245959 | 0.861661 | 0.124246 | 0.394839 | 0.23236 | 0.603531 | — | — | 0.474349 | — | 0.744979 | 0.367086 | — | — | — | — | — | — | 0.558759 | 0.488417 | — | — | 0.268258 | 0.231545 | 0.113905 | 0.10114 | — | — | — | — | — |
| 2.1.287 | 0.255694 | 0.860943 | 0.130512 | 0.398207 | 0.22398 | 0.580606 | — | — | 0.486412 | — | 0.732217 | 0.372231 | — | — | — | — | — | — | 0.570521 | 0.472712 | — | — | 0.260396 | 0.229905 | 0.112408 | 0.097659 | — | — | — | — | — |
| 2.1.288 | 0.252245 | 0.850935 | 0.114665 | 0.392857 | 0.228834 | 0.653173 | — | — | 0.48468 | — | 0.733234 | 0.380491 | — | — | — | — | — | — | 0.6001 | 0.482764 | — | — | 0.265522 | 0.226715 | 0.117025 | 0.101566 | — | — | — | — | — |
| 2.1.289 | 0.25334 | 0.909701 | 0.118378 | 0.397466 | 0.230586 | 0.613609 | — | — | 0.476427 | — | 0.758254 | 0.38755 | — | — | — | — | — | — | 0.58707 | 0.498698 | — | — | 0.270811 | 0.233209 | 0.116311 | 0.095377 | — | — | — | — | — |
| 2.1.290 | 0.264398 | 0.913671 | 0.127023 | 0.376629 | 0.240915 | 0.653922 | — | — | 0.528847 | — | 0.743275 | 0.371305 | — | — | — | — | — | — | 0.59559 | 0.493815 | — | — | 0.267004 | 0.248693 | 0.114914 | 0.106292 | — | — | — | — | — |
| 2.1.293 | 0.27512 | — | 0.128976 | 0.411107 | 0.249953 | 0.70118 | — | — | 0.539082 | — | — | 0.377414 | — | — | — | — | — | — | 0.548037 | 0.468933 | — | — | 0.265703 | 0.24093 | 0.114735 | 0.100852 | 0.012407 | 0.317412 | 0.286383 | — | — |
| 2.1.301 | 0.274533 | — | 0.129101 | 0.403438 | 0.240284 | 0.67537 | — | — | 0.517589 | — | — | 0.402355 | — | — | — | — | — | — | 0.612923 | 0.494517 | — | — | 0.280173 | 0.257248 | 0.117212 | 0.102852 | 0.013514 | — | — | — | — |
| 2.1.302 | 0.27476 | — | 0.12945 | 0.40878 | 0.245385 | 0.740242 | — | — | 0.530708 | — | — | 0.380468 | — | — | — | — | — | — | 0.574091 | 0.504586 | — | — | 0.273363 | 0.239183 | 0.115889 | 0.10331 | 0.013046 | — | — | — | — |
| 2.1.303 | 0.276517 | — | 0.137723 | 0.389024 | 0.246957 | 0.737249 | — | — | 0.523192 | — | — | 0.364898 | — | — | — | — | — | — | 0.596419 | 0.49159 | — | — | 0.287614 | 0.239036 | 0.126507 | 0.101658 | 0.012874 | — | — | — | — |
| 2.1.304 | 0.285516 | — | 0.12591 | 0.413152 | 0.241888 | 0.704236 | — | — | 0.581775 | — | — | 0.368391 | — | — | — | — | — | — | 0.588224 | 0.501371 | — | — | 0.272162 | 0.243686 | 0.118036 | 0.099702 | 0.013014 | — | — | — | — |
| 2.1.305 | 0.269313 | — | 0.128956 | 0.40663 | 0.239664 | 0.687509 | — | — | 0.540101 | — | — | 0.368287 | — | — | — | — | — | — | 0.552777 | 0.499523 | — | — | 0.275369 | 0.23456 | 0.115157 | 0.100287 | 0.01288 | — | — | — | — |
| 2.1.306 | 0.292285 | — | 0.128355 | 0.415916 | 0.240675 | 0.696166 | — | — | 0.536437 | — | — | 0.35114 | — | — | — | — | — | — | 0.594312 | 0.48451 | — | — | 0.26567 | — | 0.114957 | — | 0.013645 | — | — | — | — |
| 2.1.309 | 0.271531 | — | 0.131535 | 0.404778 | 0.246778 | 0.718432 | — | — | 0.544257 | — | — | 0.38016 | — | — | — | — | — | — | 0.61849 | 0.500838 | — | — | 0.272296 | 0.246157 | 0.118141 | 0.098034 | 0.012776 | — | — | — | — |
| 2.1.310 | 0.2726 | — | 0.126666 | 0.410776 | 0.243638 | 0.662596 | — | — | 0.534537 | — | — | 0.39158 | — | — | — | — | — | — | 0.612138 | 0.493487 | — | — | 0.283874 | 0.260408 | 0.118133 | 0.103902 | 0.013107 | — | — | — | — |
| 2.1.311 | 0.25525 | — | 0.121814 | 0.387315 | 0.241023 | 0.712077 | — | — | 0.568956 | — | — | 0.36072 | — | — | — | — | — | — | 0.599546 | 0.50011 | — | — | 0.288596 | 0.238603 | 0.122342 | 0.100448 | 0.012357 | — | — | — | — |
| 2.1.312 | 0.272963 | — | 0.129309 | 0.399972 | 0.239228 | 0.720583 | — | — | 0.54346 | — | — | 0.370936 | — | — | — | — | — | — | 0.648775 | — | — | — | 0.276535 | — | 0.118213 | — | 0.013254 | — | — | — | — |
| 2.1.318 | 0.285755 | — | 0.120486 | 0.394122 | 0.241499 | 0.709363 | — | — | 0.527107 | — | — | 0.384219 | — | — | — | — | — | — | 0.561625 | 0.486796 | — | — | 0.281136 | 0.227097 | 0.11595 | 0.097665 | 0.012456 | — | — | — | — |
| 2.1.321 | 0.272182 | — | 0.125245 | 0.438731 | 0.241937 | 0.651376 | — | — | 0.548398 | — | — | 0.366116 | — | — | — | — | — | — | 0.59697 | 0.504014 | — | — | 0.264678 | 0.238846 | 0.116889 | 0.097395 | 0.012379 | — | — | 0.527394 | 0.348641 |
| 2.1.322 | 0.259945 | — | 0.12616 | 0.439888 | 0.239271 | 0.672562 | — | — | 0.529529 | — | — | 0.369822 | — | — | — | — | — | — | 0.595599 | 0.481277 | — | — | 0.279134 | 0.243456 | 0.119007 | 0.099649 | 0.01294 | — | — | 0.472055 | 0.353539 |
| 2.1.323 | 0.278131 | — | 0.123447 | 0.40784 | 0.238585 | 0.699652 | — | — | 0.523971 | — | — | 0.376634 | — | — | — | — | — | — | 0.590615 | 0.492405 | — | — | 0.285388 | 0.234916 | 0.120937 | 0.101051 | 0.013744 | — | — | 0.584363 | 0.357154 |
| 2.1.324 | 0.264695 | — | 0.12685 | 0.420963 | 0.244583 | 0.713214 | — | — | 0.49357 | — | — | 0.372694 | — | — | — | — | — | — | 0.622137 | 0.514318 | — | — | 0.278203 | 0.249188 | 0.118546 | 0.100307 | 0.013532 | — | — | 0.442838 | 0.336239 |
| 2.1.325 | 0.274011 | — | 0.130489 | 0.436478 | 0.241792 | 0.719755 | — | — | 0.482108 | — | — | 0.374326 | — | — | — | — | — | — | 0.63244 | 0.504899 | — | — | 0.295232 | 0.233659 | 0.11875 | 0.100323 | 0.013501 | — | — | 0.431351 | 0.34793 |
Data table
| Metric | fable-5-1 (high) | fable-5-2 (high) | haiku-4-5 | haiku-5-5 | opus-4-8 (high) | opus-5 (high) | opus-5-5 (high) | sonnet-5 (high) | sonnet-5-5 (high) |
|---|---|---|---|---|---|---|---|---|---|
| Cost per session (USD)mean | 0.087124 | 0.059442 | 0.016069 | 0.001626 | 0.060247 | 0.089925 | 0.038117 | 0.034126 | 0.016088 |
| Cost per session (USD)median | 0.086868 | 0.049167 | 0.01657 | 0.001463 | 0.061961 | 0.07752 | 0.034162 | 0.035393 | 0.014953 |
| Output tokens | 605 | 503 | 745 | 938 | 484 | 778 | 513 | 431 | 414.5 |
| Input tokens | 66 | 6 | 26 | 8 | 6 | 8 | 6 | 8 | 6 |
| Cache-read tokens | 56479 | 55181 | 65926 | 67865 | 50928 | 73887 | 51373 | 114490 | 51082.5 |
| Cache-write tokens | 1130 | 670.5 | 1320 | 1381 | 809 | 1729 | 767.5 | 958 | 0 |
| Input + cache tokens | 57538 | 55902 | 66655 | 72916 | 51305 | 75701 | 51896.5 | 117618 | 51406 |
| API calls | 3 | 3 | 3 | 4 | 3 | 4 | 3 | 4 | 3 |
| Tool calls | 2 | 2.5 | 2 | 4 | 2 | 3 | 2 | 3 | 3 |
| Thinking blocks | 0 | 0 | 3 | 1.5 | 1 | 1 | 0 | 1 | 0 |
| Latency (s, wall-clock) | 17.959 | 10.5705 | 8.024 | 6.1705 | 8.953 | 11.833 | 6.3935 | 8.641 | 6.629 |
| Suite cost (USD) | No published column carried Suite cost (USD) for this cycle. | ||||||||
Data table
| Metric | fable-5-1 (medium) | fable-5-2 (medium) | opus-4-8 (medium) | opus-5 (medium) | opus-5-5 (medium) | sonnet-5 (medium) | sonnet-5-5 (medium) |
|---|---|---|---|---|---|---|---|
| Cost per session (USD)mean | 0.067448 | 0.047329 | 0.047597 | 0.065308 | 0.030791 | 0.029081 | 0.013616 |
| Cost per session (USD)median | 0.059721 | 0.040754 | 0.044672 | 0.061105 | 0.026372 | 0.031772 | 0.009508 |
| Output tokens | 480 | 402 | 442 | 500.5 | 441 | 394 | 344 |
| Input tokens | 66 | 6 | 6 | 6 | 6 | 8 | 6 |
| Cache-read tokens | 56471 | 51747 | 50858 | 54210 | 47694 | 116947 | 50904 |
| Cache-write tokens | 809 | 0 | 414 | 1038 | 464 | 845 | 0 |
| Input + cache tokens | 57227 | 55369 | 51305 | 55283.5 | 51616 | 117534 | 51218 |
| API calls | 3 | 3 | 3 | 3 | 3 | 4 | 3 |
| Tool calls | 2 | 2 | 2 | 2 | 2 | 3 | 2 |
| Thinking blocks | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| Latency (s, wall-clock) | 16.598 | 9.921 | 6.978 | 7.633 | 5.695 | 7.895 | 5.507 |
| Suite cost (USD) | No published column carried Suite cost (USD) for this cycle. | ||||||
Claude Code CLI 2.1.325 Analysis
Measured 2026-11-14 14:00 UTC from this cycle’s benchmark runs, written up from those measurements alone, then checked by an independent reviewer. Where the wording and the tables disagree, the tables are correct.
Verdict
A task at the high reasoning setting costs this, per session, cheapest to most expensive:
- haiku-5-5: $0.0016
- haiku-4-5: $0.0161
- sonnet-5-5: $0.0161
- sonnet-5: $0.0341
- opus-5-5: $0.0381
- fable-5-2: $0.0594
- opus-4-8: $0.0602
- fable-5-1: $0.0871
- opus-5: $0.0899
At the medium setting, the same per-session costs:
- sonnet-5-5: $0.0136
- sonnet-5: $0.0291
- opus-5-5: $0.0308
- fable-5-2: $0.0473
- opus-4-8: $0.0476
- opus-5: $0.0653
- fable-5-1: $0.0674
Across the last 10 cycles at the high setting, each family’s direction and the dollar band it stayed inside:
- opus-4-8 drifted up, within $0.0536 to $0.0602
- fable-5-1 no clear direction, within $0.0747 to $0.0871
- haiku-4-5 no clear direction, within $0.0147 to $0.0165
- haiku-5-5 no clear direction, within $0.0015 to $0.0016
- opus-5 no clear direction, within $0.0764 to $0.0916
- opus-5-5 no clear direction, within $0.0351 to $0.0397
- sonnet-5 no clear direction, within $0.0312 to $0.0348
- sonnet-5-5 no clear direction, within $0.0157 to $0.0164
- fable-5-2 too few cycles to say, within $0.0594 to $0.0755
Across the last 10 cycles at the medium setting:
- fable-5-1 no clear direction, within $0.0362 to $0.0678
- opus-4-8 no clear direction, within $0.0460 to $0.0509
- opus-5 no clear direction, within $0.0632 to $0.0713
- opus-5-5 no clear direction, within $0.0094 to $0.0324
- sonnet-5 no clear direction, within $0.0286 to $0.0297
- sonnet-5-5 no clear direction, within $0.0134 to $0.0141
- fable-5-2 too few cycles to say, within $0.0473 to $0.0488
This cycle has no percentage move to report at either setting. At the high setting, no family has sessions on both sides of the comparison with the previous CLI version, so no change could be measured; the same is true at the medium setting. Nothing here says costs held steady, only that this cycle’s runs cannot be set against the previous version’s.
One other thing: the spread between families is enormous. Comparing haiku-5-5 with opus-5 at the high setting on identical prompts, opus-5 costs 5430.7% more, and that gap is statistically significant.
Which cost figure is which – setting high
No family has sessions on both sides, so there is nothing to reconcile.
Which cost figure is which – setting medium
No family has sessions on both sides, so there is nothing to reconcile.
What these terms mean
New to this report? Here is the vocabulary used below, in plain words. Skip ahead if you already know it.
- Session (also “task”): one complete run of one prompt by one model, start to finish. Every cost in this report is per session.
- Family: a model line, such as opus or sonnet. One family can cover more than one version, for example opus-4-8 and opus-5.
- Cache, warm and cold: the tool reuses recent context instead of sending it again, which is far cheaper. A session that gets to reuse it is “warm”; one that has to send everything fresh is “cold”. Whether a given session lands warm or cold is partly luck, and that luck moves the cost.
- Cache luck (or cache-hit luck): the cost swing caused by how many sessions happened to land warm rather than cold, rather than by any real change in the tool or the model.
- Blended cost: cost measured across every session, warm and cold alike. This is the real money figure, and it is what the cost feed shows. “Blended” says which sessions are counted, not how they are averaged; see “Which cost figure is which” above for the averaging.
- Warm-only cost: cost measured across warm sessions only. Removing the cold ones removes most of the luck, so this is the fair test of whether cost actually changed.
- Pooled: all sessions for a family added together into a single number, rather than split out per prompt.
- Mean cost per session: the ordinary average across every session. It multiplies out to real spend, so it is the budgeting figure and the one this report leads with.
- Median cost per session: the middle session once they are lined up by cost. It ignores an unusually cheap or expensive run, but it does not add up to real spend.
- Suite cost: the price of one full pass over every prompt in the catalog, found by taking each prompt’s median and adding those up. It is a total for the whole catalog, not an average per session, so it is a larger number than either figure above and is not comparable to them.
- Cell: one family and one prompt combination, for example haiku on the rename-refactor prompt. Cells hold few sessions, so a single cell is noisy.
- n: how many sessions sit behind a number. A larger n is more trustworthy.
- p-value, and “significant”: the same prompt run twice costs different amounts, so a p-value asks whether two sets of runs can be told apart at all. Line up every run from both cycles: if one cycle’s runs sit consistently higher, p is near 0 and the difference is called “significant”; if the runs are mixed together, p is near 1 and the two sets look like the same thing measured twice. Precisely, it is the chance of seeing a gap at least this lopsided if the two cycles were truly identical. A high p is not weak evidence of a change, it is no evidence either way.
- API-equivalent cost: dollars calculated at published API rates. It is not necessarily dollars billed, because a subscription plan may already cover the usage.
Pooled blended comparison
This is the headline lens: the real per-task spend in dollars per session, counting every session in the cycle, including sessions that paid the cost of a cold cache. It is what a user actually pays.
Setting high, comparing nightly-2.1.323 with nightly-2.1.325:
No family has sessions on both sides – no comparison possible.
Setting medium, comparing nightly-medium-2.1.323 with nightly-medium-2.1.325:
No family has sessions on both sides – no comparison possible.
Each setting is compared only against its own previous run at the same setting, and at both settings the overlap needed for that comparison is missing this cycle.
Pooled warm comparison (significance control)
This is the control that answers whether a cost change is believable. It looks only at sessions that ran with a warm cache, which removes the random luck of which sessions happened to start cold, and then compares the individual run costs from each cycle to see whether the two sets can be told apart.
Setting high:
No family has sessions on both sides – no comparison possible.
Setting medium:
No family has sessions on both sides – no comparison possible.
With no blended move to test at either setting, there is nothing for this control to confirm or fail to confirm this cycle.
What the reasoning setting costs
This is the one place the two settings are measured against each other: the blended mean dollars per session at medium set against the same model at high, with the individual run costs from the two pooled session sets compared to test the difference. A negative percentage means medium is cheaper.
| model | high mean $ | medium mean $ | change | p | n |
|---|---|---|---|---|---|
fable-5-1 |
0.0871 | 0.0674 | -22.6% | 0.052 | 61/53 |
fable-5-2 |
0.0594 | 0.0473 | -20.4% | 0.298 | 54/53 |
opus-4-8 |
0.0602 | 0.0476 | -21.0% | 0.039 | 61/53 |
opus-5 |
0.0899 | 0.0653 | -27.4% | 0.006 | 61/60 |
opus-5-5 |
0.0381 | 0.0308 | -19.2% | 0.250 | 54/53 |
sonnet-5 |
0.0341 | 0.0291 | -14.8% | 0.039 | 61/53 |
sonnet-5-5 |
0.0161 | 0.0136 | -15.4% | 0.201 | 54/53 |
Medium is the cheaper setting for every model measured at both. Three of those differences clear the 0.05 line:
- opus-4-8 is 21.0% cheaper at medium, p=0.039
- opus-5 is 27.4% cheaper at medium, p=0.006
- sonnet-5 is 14.8% cheaper at medium, p=0.039
For the other four, the difference cannot be told apart from normal run-to-run variation:
- fable-5-1 is 22.6% cheaper at medium, p=0.052
- fable-5-2 is 20.4% cheaper at medium, p=0.298
- opus-5-5 is 19.2% cheaper at medium, p=0.250
- sonnet-5-5 is 15.4% cheaper at medium, p=0.201
Something may well be different for those four; this measurement just cannot see it. Two models, haiku-4-5 and haiku-5-5, were measured at one setting only, so they carry no cross-setting figure.
Notable cells (secondary)
This lens looks for individual prompts that moved more than the rest. At both settings, zero cells were available to compare, because no family had sessions on both sides of the version comparison, and no cell moved beyond the noise threshold. There is nothing to watch from this cycle. When movers do appear here they are single prompts with roughly five runs each, tested against many other prompts at once, so a few will always look extreme by chance; they are a prompt to re-examine next cycle, never a result on their own.
Caveats
Two cost lenses exist, and they answer different questions. The blended figure is the real per-task spend, counting every session including the ones that paid for a cold cache, and it is the headline that matches the dashboard. The warm-only figure strips out that random cache luck and is the control for whether a change is real. Judge “did cost change” by the warm comparison, not by the blended percentage: a blended move with no significant warm move may be cache luck rather than a real change.
The compared cycles ran on different CLI versions, 2.1.323 and then 2.1.325. Any pooled difference therefore measures the effect of the CLI version, which is what this benchmark exists to measure. Do not read it as a statement about the model itself without a separate test that holds the CLI version fixed.
Token capture for the Haiku models is known to be incomplete, so treat Haiku cost and token figures as lower bounds, not exact amounts.
Timings are wall-clock and include the time the model spends thinking. Costs are calculated as the API-equivalent price of the work done, not dollars actually billed by a provider.
Verification appendix
- Verifier verdict: PASS after 1 round(s).