Your default AI model may be one tier too big

Anthropic reports that its small model, Claude Haiku 5.5, scores 72.4% (partial credit) on an offline computer-use test. Its predecessor scored 15.7%. Per token, Haiku 5.5 costs 90% less than its predecessor for prompts up to 100k tokens.

Two cautions come with those numbers. They are the vendor’s own figures, on one benchmark. And “partial credit” means a task can score without being completed in full. Neither point makes the result unimportant, but both tell you what it can support. It supports a decision to test. It does not support a decision to switch.

If a job does not need the larger model, paying for it buys you nothing you can see. A larger model is slower and costs more per token, and on routine work its extra capability goes unused. Because the cost is spread across thousands of small calls, nobody sees it as a decision. It looks like the price of using AI.

The reverse mistake is also real. For work where accuracy decides the outcome, I would not look at price first. In a critical process, I am not worried about cost, I am worried about accuracy. The aim is the cheapest model that does each task well enough, not simply the cheapest model.

So the question is never “big or small”. It is “which tier does this particular job need, and how do I know?”

The original post pointed to a set of slides with four steps for re-testing your own workloads. I have not reproduced those slides here, so what follows is my own version of the method rather than a transcript of them.

1. List the jobs, not the tools. Write down each distinct task the model does for you: classify an enquiry, extract fields from an invoice, draft a reply, summarise a report. Each one can sit on a different tier.

2. Build a small test set from real work. Take actual past inputs and, for each, the output you consider correct. Include the awkward cases, because a small model usually fails on those first. The set does not need to be large to be useful, but it must be yours. A vendor’s benchmark tells you about the vendor’s tasks.

3. Run the smaller tier and score it the way you would score a person. Compare against your correct outputs. Count the errors that matter separately from the errors that do not. A wrong label on a high-stakes record is not the same as an awkward phrase in a draft.

4. Decide per job, and write it down. Move the jobs where the smaller tier is good enough. Keep a larger model where it is not, or where a mistake is expensive. Record what you found and when, because a result that is true today may not hold after the next model release.

This fits an approach I use for cost control, which is to assign each kind of work to a different tier. The most capable model plans and designs, used sparingly. A strong but cheaper model builds. Lighter models do repetitive work such as searching, formatting and extraction. For tasks that repeat thousands of times, a small open model on your own machine can take the volume.

The thing I would stress is that the layers are decided by testing, not by assumption. “Sonnet is enough to do a web search” is a judgement I made from use. It is not a rule that holds for every firm’s data.

The 72.4% figure also tells you the other half of the story. Partial credit on a test where the earlier model scored 15.7% means the new one is far better, and also means it does not complete every task. Anything that depends on the model finishing a long sequence without a slip needs checking.

I keep two safeguards in mind. First, put a check on the output where an error is costly, either a person or a stronger model. Second, keep the harder, rarer cases on the larger tier even if the routine ones move down. The saving usually comes from the routine bulk, not from the exceptions.

Suppose you have two kinds of job. One is a high-volume tagging task where mistakes are cheap to catch. The other is a monthly analysis that goes to a board. For the first, the case for the smaller tier is strong: the task repeats, the output has a fixed shape, and you can spot-check. For the second, volume is low, so the cost saving is small, and the cost of an error is large. The reasoning points in opposite directions, and a single default cannot reflect both.

That is the whole argument in miniature. A default is a decision made once, for everything. The evidence usually supports a decision made job by job.

Pick the one AI job in your organisation that runs most often. Build a test set of fifty or so real examples with the correct outputs. Run it on the next tier down. If it holds up, you have a saving. If it does not, you have a measured reason to keep paying for the larger model, which is also useful.

Which of your AI jobs would you try on a smaller model first?