The Pareto frontier for AI models

Pareto efficiency is a useful concept that helps you avoid unnecessary tradeoffs.

When we're deciding on which model to choose for a given task, there are lots of different options we can choose from. We might be willing to pay any amount for the most intelligent model; we might be OK with a cheap model that’s not that smart; or we might want to land somewhere in the middle. But what we basically never want to do is to pay more for a model that’s as smart as a cheaper one. That would just be a waste of money without any benefit.Of course this assumes that we can accurately benchmark what 'smart' means, which is easier for some tasks (coding) than others (writing)

This is what tracking models’ Pareto efficiency allows us to do. If we chart model cost versus model intelligence, then we can see that while there are many models, we actually only need to care about a small number of them, since the vast majority are not Pareto optimal. The set of models that are Pareto optimal at a given intelligence/cost level is the ‘Pareto frontier’.

Concretely, if we hover this chart we can see that we basically never want to choose GPT 5.4 mini over DeepSeek v4 Flash, since it’s the same intelligence level but 10x the cost - we'd trade off on cost without deriving a benefit in terms of intelligence. On the other hand, we might well want to consider using GLM 5.2, since in that case we'd get a meaningful intelligence boost in exchange for paying more.That’s a little bit of an oversimplification in practice: we can’t boil model intelligence down to just a single variable, and Pareto optimality depends on the benchmark we’re charting against. It might be that if we were tracking multimodal intelligence on the Y-axis, GPT 5.4 mini would perform more favourably. But regardless, my point is that we’d still want the Pareto optimal choice for that benchmark.

Considering both cost and intelligence in this way helps explain why token consumers make the choices they do. Looking at the OpenRouter leaderboard, I was initially confused as to why DeepSeek v4 Flash is so popular, since it isn't the best-performing OSS model:

But if you take another look at the chart above we can see that it bends the Pareto frontier far more than any other model: it’s much more intelligent than the other models at its price point, and much cheaper than the other models at its intelligence point. Fable might be the heavyweight champion but DeepSeek have got the pound-for-pound #1 - for now, anyway.

A good heuristic for whether we should care about a new model release or not is whether it moves the Pareto frontier.

For example, Tencent just released a new model, Hy3. We can see from the model page on HuggingFace that it represents a modest benchmark increase over the previous version, ‘Hy3-preview’. But it’s not obvious that we should care about the model, since there are other models that perform significantly better - the new mode has 71.7 on Terminal Bench 2.1, noticeably less than GLM 5.2’s score of 81.0.

However, if we plot Hy3 on our chart we can see that it has a strong chance of bending the Pareto frontier at its price point: it scores 10% higher on Terminal Bench than DeepSeek v4 Flash, while costing basically the same,I am assuming that the inference costs for Hy3 will be the same as the current OpenRouter costs for Hy3-Preview. I think that’s a reasonable assumption since HuggingFace shows identical numbers of parameters, activated parameters, layers etc. staking its own claim on the P4P #1 spot:

And while it’s true the model isn’t quite as intelligent as GLM 5.2, it’s much cheaper - about 14x cheaper per-token! The benchmarks, which only show intelligence and not cost, don’t tell the full story.Plotting cost against intelligence doesn't quite tell the full story either - the primary limitation is that this 2d chart doesn't tell us how many tokens the models are spending to achieve this level of intelligence. In practice, GLM 5.2 for instance has been known to spend about twice as many tokens as Sonnet 5 for to achieve a similar benchmark results. I hope to write about this in a future post.

There are a few existing sites that allow you to track the Pareto frontier for models, but I found that they were all missing something for my use case:

• LMArena track the pareto frontier for Arena ELO, but that’s not a good benchmark
• Some vibecoded public options (Model Depot and LLM Pareto Frontier) also track Arena ELO, same problem, and are out of date
• Artificial Analysis have a lot of good data, but they don’t display Pareto frontier, miss some new releases, and their cost data is sometimes wrong since they report a single cost number based on the official provider API rather than the current cheapest OR choice

So I’ve created ai-pareto.vercel.app, which is the site used to generate the charts above. It displays the true pareto frontier based on trustworthy benchmarks (AA intelligence index or Terminal Bench 2.1 scores) and the actual cheapest option for tokens (based on OpenRouter data). Check it out! I’ve added a cron job to pull new model additions so it will stay up to date, and I may add some additional models manually as I did with hy3 above.