Are the Artificial Analysis benchmarks really worthwhile any more?
They really don't seem to match my real-world experience, and based on the comments I see I don't think that match most other people's either.
For example, Opus 5 was at the top for some time. My experience is that it's not noticeably better than Opus 4.8, and it definitely seems worse than Fable 5, which AA benchmarks put behind Opus 5.
GPT 5.6-sol and Opus 5 seem pretty interchangeable, although Sol is noticeably better at finding problems in code, particularly edge cases.
Well, seems like ECI [0] and the AA index is diverging quite a bit. Benchmarking LLM is tough and I think we are seeing the limitations of current benchmarks and applicability to real tasks.
I believe the development of new models has been like an s-curve: There were enormous incremental improvements earlier on, but now we're reaching the right-hand side of the s-curve and all the new models are clustering together.
Even the lighter and smaller and cheaper models going to end up near that limit, and that will likely evaporate the perceived economic value of both OpenAI and Anthropic unless they manage to lock it in/offset it with platform effects and branding (they might well be able to do that).
My impression is that we might well be able to move past that limit, but we would need another radical invention like the transformer architecture that the whole current generation of models is built on.
Fascinating. This is the only benchmark I've seen so far with lack-luster results. I don't understand enough about AA's specific methodology to get the implications.
In the general Intelligence Index it scores exactly equal to Sol (61). In the Agentic Index it scores significantly lower than Sol (51 vs 58).
In both it scores lower than Fable 5.1, Opus 5 and even Muse Spark 1.3.
Am I missing something or is this not looking too... stellar?
And yet I(and many others publicly on x) chucked Claude for codex back in June. Codex is my workhorse. The main reason I use codex is because of its language. I just got really tired of reading long weirdly worded prose that I had to fix with some skill(though some people like matt pocock and dex horothy have good ideas on this, buts it's just wasteful). This is triply bad for learning newer stuff because it goes up and down the abstraction layer on any topic like mad. One moment it would be explaining a high level detail and then cite contrasts and then point indirectly to an implementation detail as an example. It's idea of explaining more abstractly was also weird in a different way.
Fable was better but I don't have 1000$/day to spend on it.
Are the Artificial Analysis benchmarks really worthwhile any more?
They really don't seem to match my real-world experience, and based on the comments I see I don't think that match most other people's either.
For example, Opus 5 was at the top for some time. My experience is that it's not noticeably better than Opus 4.8, and it definitely seems worse than Fable 5, which AA benchmarks put behind Opus 5. GPT 5.6-sol and Opus 5 seem pretty interchangeable, although Sol is noticeably better at finding problems in code, particularly edge cases.
I have no faith in these benchmarks. Muse 1.3 shouldn’t even be in the same conversation yet it scores above GPT 5.6 & 6.0
Well, seems like ECI [0] and the AA index is diverging quite a bit. Benchmarking LLM is tough and I think we are seeing the limitations of current benchmarks and applicability to real tasks.
[0]: https://x.com/EpochAIResearch/status/2095602754282783108
Title: "major gains"
First chart: from score 61 (GPT-5.6 Sol) to drumroll 61 (GPT-6 Astra)
Indeed. Though to be fair it is referring to "Artificial Analysis Coding Agent Index", from 65 to 67.
I think they mean cost per task, where Astra is now on the Pareto frontier.
I don't understand how those labs are releasing models so close in performance to one another?
Are they just scaling more? getting more data at the same rate? training against the same benchmarks? making the same breakthroughs?
How can this be explained?
We're hitting a limit, that's what's happening.
I believe the development of new models has been like an s-curve: There were enormous incremental improvements earlier on, but now we're reaching the right-hand side of the s-curve and all the new models are clustering together.
Even the lighter and smaller and cheaper models going to end up near that limit, and that will likely evaporate the perceived economic value of both OpenAI and Anthropic unless they manage to lock it in/offset it with platform effects and branding (they might well be able to do that).
My impression is that we might well be able to move past that limit, but we would need another radical invention like the transformer architecture that the whole current generation of models is built on.
That make sense, s-curve explains it I think. Thanks.
Fascinating. This is the only benchmark I've seen so far with lack-luster results. I don't understand enough about AA's specific methodology to get the implications.
It's sub-Fable 5 on mirror code: https://epoch.ai/benchmarks/mirrorcode?view=graph&tab=leader...
In the general Intelligence Index it scores exactly equal to Sol (61). In the Agentic Index it scores significantly lower than Sol (51 vs 58). In both it scores lower than Fable 5.1, Opus 5 and even Muse Spark 1.3.
Am I missing something or is this not looking too... stellar?
And yet I(and many others publicly on x) chucked Claude for codex back in June. Codex is my workhorse. The main reason I use codex is because of its language. I just got really tired of reading long weirdly worded prose that I had to fix with some skill(though some people like matt pocock and dex horothy have good ideas on this, buts it's just wasteful). This is triply bad for learning newer stuff because it goes up and down the abstraction layer on any topic like mad. One moment it would be explaining a high level detail and then cite contrasts and then point indirectly to an implementation detail as an example. It's idea of explaining more abstractly was also weird in a different way.
Fable was better but I don't have 1000$/day to spend on it.
more like 5.7 not 6
5.6.1
It’s based on a new and larger pre train if I understand correctly, hence the major version.