Qwen 3.8-Max and Claude Opus 5 show why raw benchmark scores don't predict the bill

Claude Opus 5 show why raw benchmark scores don't predict the bill">

Alibaba released Qwen 3.8-Max this week and marketed the preview as second only to Claude Fable 5 (their launch-day table was more equivocal: the model leads on one of 12 coding-agent rows). But an independent harness came close to the opposite conclusion: a benchmark run , apparently using the Preview version, put Qwen 3.8-Max's best effort setting mid-pack, and its default setting last. Both results are real and defensible. The gap between them is about token and time budgets, and that matters because those figures aren’t usually headline numbers. Alibaba's footnotes give its coding numbers a five-hour timeout, and up to 12 hours per run on PaperBench. The independent harness, VulcanBench, allowed between 45 and 60 minutes of wall clock time . A time budget between five and 16 times larger on Alibaba’s side explains the huge difference in results. It’s time to do two things to start accounting for these differences when choosing models. First, the metric to use is cost per successful task: total spend, including everything you spent on attempts that failed, divided by the tasks that actually passed your acceptance check. Second, you need to make time or token budgets an explicit part of your acceptance criteria, not a hidden detail. Price per token has stopped predicting the bill The comparison everyone published in Qwen 3.8-Max's first week was a price comparison, because that was the only data available. It is not a cheap model. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, lists at 14 cents per million input tokens and 28 cents output . Qwen 3.8-Max lists at $2 and $6. Kimi K3 sits at $3 and $15. Those prices tell you less than they used to, for a reason specific to reasoning models like Qwen: getting to a result costs thinking tokens. A model that spends most of its token allowance on reasoning can reach a token cap before it writes the answer, giving you an empty result indistinguishable from a total failure at the cost of a full run. Artificial Analysis has the cleanest published measurement of how this can affect real agent spend: running its Intelligence Index on DeepSeek-V4-Flash at maximum effort took 210 million output tokens against a class median of 100 million. Absolute cost stayed low anyway, because the tokens were so cheap. But verbosity costs time, not just money, and depending on your use case that can sink you. What you need is a number that counts everything you spent, including the attempts that came back empty, against the tasks that actually got done in the time and token budget you specified. This is what a cost-per-success metric helps you see. Your failure rate is partly a configuration setting A run that produces a wrong answer and a run that runs out of budget are different events with different fixes. Almost no harness distinguishes them, and almost no leaderboard reports the split. I hit this building an agent benchmark of my own : the harness logged a failure and nothing about why, and I had to add the distinction myself. When you do separate them, budget exhaustion turns out to dominate. Long-Horizon-Terminal-Bench , published in July, ran 17 frontier models across 46 tasks through a shared harness with one 90-minute attempt each. Timeouts accounted for 79% of unresolved runs, against 19% for agents that stopped on their own and 3% for harness errors. The authors are careful about what that does and does not mean: the timed-out runs were not close to finishing, with mean reward between 0.10 and 0.35, so you cannot assume more time would have resulted in success. But the lesson is: benchmarks are implicitly measuring time efficiency, whether or not they shout about that. The clearest published example of the mechanism comes from VulcanBench, the same open-source harness behind the Qwen chart. In a report dated July 26 , Claude Opus 5's lowest-effort setting was its best, solving 20 of 23 tasks against 18 at high effort. The extra reasoning wasn’t useless: high effort returned the fewest wrong answers of any setting, one against three. It ran out of clock instead, and a timeout scores zero. Two of its three regressions were cutoffs on tasks that low effort solves, and given unlimited time on both it only ties its cheapest setting, at 3.1 times the cost. That has a direct consequence for anyone building a routing ladder. The standard design escalates to more reasoning when a cheap attempt fails, on the assumption that the next rung is better and merely costs more. For a meaningful share of model and task combinations that assumption is wrong, and you pay the higher rung's price to escalate into a timeout or hitting a cap. Who is already measuring this Several groups have landed on cost per successful task independently in the last few months, which is the strongest signal it's becoming standard. VulcanBench reports dollars per solved task as a headline column and has since its earliest reports . Long-Horizon-Terminal-Bench publishes per-task cost next to accuracy, and its most instructive row is GPT-5.4 at roughly $26 per task with a much lower pass rate than Grok 4.5 at about $11. TestEvo-Bench runs agents under a cost cap, and Claude Code's test-generation score falls from 71% to 44% at the tighter cap. Vendors are already on board with the idea of measuring per successful task. HubSpot moved its Breeze Customer Agent in April to 50 cents per resolved conversation , down from $1 per handled conversation. Zendesk bills per automated resolution . Fin charges 99 cents per outcome and bills only on end-to-end resolution. What to change this week Emit a failure reason on every agent run as a required field, with budget exhaustion, verifier failure and harness error as distinct values rather than one failure flag. Until you can separate a timeout from a wrong answer, your pass rate is measuring two things at once and you cannot tell which one to fix. Compute cost per successful task per effort level, not just per model. Total spend including failed attempts, divided by tasks that passed your acceptance check. The ranking will not match the rate card, and the cheapest setting may well win. Cap on tokens rather than wall clock unless latency is genuinely in your service level objective. A wall-clock cap scores your provider's serving speed as model quality. Check the default effort setting on everything you have deployed. Qwen 3.8-Max runs at its highest reasoning setting when the effort field is unset , and its highest setting was its worst performer in independent testing. A team that never touches that parameter is running the configuration that costs the most per solved task.