i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof
They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...
They have lowered the price several times; and often released more for the same price. But it is normal to consider the best ever price to performance the benchmark and anything below that disappointing.
Realistically it can't match Fable, let alone Opus today. It can however beat basically all of the models from 1 year ago. If last years state of the art is "good enough" for you today then local is great.
Reasonable throughput depends a lot on your hardware. I'm running Qwen3.8-Flash-Next on a 128GB Strix Halo box at 50t/s gen (using the gufo inference engine) and it feels like Opus4.6/4.8 level performance. It's an amazingly good model. I've got it coding a C->microcode compiler in OCaml and it's doing great (there's also verilog code generation involved as well so this is a pretty good stress-test, I think).
Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.
Maybe at Opus 4.6 level for the "easier" tasks. But honestly if you were given a choice between a computer with Qwen 3.8 or one with even older Sonnet 5, both local, same tok/s, window size, etc... what are you picking?
I've got QFN making a C(sub/superset)->microcode compiler, writing it in OCaml. It's come up with optimizations that Claude didn't come up with (in the 4.6/4.8 era) when it was making the previous C-based implementation. The compiler generates microcode for an algorithmic state machine/transducer in addition to Verilog. I've been working with QFN on this for about 4 days now and it's pretty close to feature complete - yesterday it was able to compile our Game of Life C implementation and the microcode ran on a Tang Nano 9K FPGA board with LCD displaying the cells. I do not think this is an "easier" task.
100% agree. A local model might not have the same stretch goal capabilities, but at least I don't have to constantly second guess myself in terms of the inconsistencies I think I'm noticing.
No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.
Local AI is the way forward. Using someone else's harness and someone else's model is just borrowing a workflow that's liable and likely to change on a whim.
This account claims Sonnet and Opus 5.6 are in A/B testing now. There is nerfing when they release a new version, for reallocating resources.
https://x.com/tokengremlin/status/2119783808773841101
how reliable is this twitter account?
i've seen people on social media claiming all sorts of things about Fable 5.5, Opus 5.6 and Sol 6.2 and what-not for the past few weeks, but it just seems like engagement farming with no proof
https://www.ueddit.com/r/ClaudeCode/comments/1xr9x5y/the_ner...
They claim they've used the same cli version, but don't publish their methodology (maybe I've missed it), eg how do I know they didn't accidentally turn on claude code auto update after all? At least https://github.com/ninjahawk/livenerf seems unchanged so far and is much more thorough...
And here I thought it was them taking away datacenter usage so they could drop a new model soon.
Do they ever test by giving the model more resources to see if they can give users more for the same price ? Just saying
They have lowered the price several times; and often released more for the same price. But it is normal to consider the best ever price to performance the benchmark and anything below that disappointing.
You know what will never ab test your usage? Local ai. Just saying.
Yep, this.
And with Strata running at 50tok/s on my desktop is all I need. Would I like faster? Sure! But it's super useful in all sorts of tasks now.
I tried it. Nothing local or open weights can touch Fable/Opus/Gemini for my use cases.
What it did you try, and what are your use cases? Genuinely curious.
I am also curious what exactly you tried. Depending on your hardware it might also be worthwhile to stay in-tune with Qwen 4 and GLM 5.5/6.0 leaks.
Realistically it can't match Fable, let alone Opus today. It can however beat basically all of the models from 1 year ago. If last years state of the art is "good enough" for you today then local is great.
What open-weight models can match Opus 4.x or GPT 5.4 performance at a reasonable throughput?
Reasonable throughput depends a lot on your hardware. I'm running Qwen3.8-Flash-Next on a 128GB Strix Halo box at 50t/s gen (using the gufo inference engine) and it feels like Opus4.6/4.8 level performance. It's an amazingly good model. I've got it coding a C->microcode compiler in OCaml and it's doing great (there's also verilog code generation involved as well so this is a pretty good stress-test, I think).
Agreed.
Flash Next is really great, but you still need pretty decent hardware. A DGX Spark, Strix Halo, or Mac with a lot of memory. You can use things like Strata to squeeze it into a system with only 32GB of VRAM by offloading a lot of it to system memory and still get something that "works", but the quality falls off a cliff too. At that point you get better results out of Qwen 3.8 27b in int4.
Never go below a four bit quant if you want good results. A worse quant of a bigger model is often easily beaten by the small model at decent quality. A lot of the people who think local AI suck think that because they don't understand that low quants == dumbass model. You aren't just compressing data, you're reducing the amount of compute it uses which effectively lowers it's IQ.
If you want to run something actually useful for cheap Qwen 3.8 27b on a 32GB GPU in int4 should be passable. It's more like Sonnet for a year ago. To match last years Opus you really need at least 128Gb.
Personally I have two Sparks connected with ConnectX 7. Now I can run GLM 5.3 locally at 30-40 tps and it's honestly been great. It really cut my dependence on cloud providers and I can run an abliterated version that never refuses my Cyber related tasks.
From experience, qwen3.8 27b with gsq/rco at iq3_xxs has good performance with little intelligence loss.
Some of the weird new quants really do push the limits, but at the end of the day that compute you're eliminating is the actual 'work' the model is doing.
Folks like to think of 'weights' as just storage, but they aren't (except with stuff like engrams). Those digits of precision encode actual computation, not just storage. There is always a cost, it's lossy compression after all. It's like MP3 vs AAC. If you can't hear the difference it just means you haven't encountered it YET.
That's not to say that you ever will. The cost could be in weird far out edge cases, but whatever it once encoded... It's gone now and it ain't coming back.
I'd argue that Qwen3.8-Flash-Next is basically at Opus 4.6/4.8 level from about 6 months ago.
Maybe at Opus 4.6 level for the "easier" tasks. But honestly if you were given a choice between a computer with Qwen 3.8 or one with even older Sonnet 5, both local, same tok/s, window size, etc... what are you picking?
> Maybe at Opus 4.6 level for the "easier" tasks.
I've got QFN making a C(sub/superset)->microcode compiler, writing it in OCaml. It's come up with optimizations that Claude didn't come up with (in the 4.6/4.8 era) when it was making the previous C-based implementation. The compiler generates microcode for an algorithmic state machine/transducer in addition to Verilog. I've been working with QFN on this for about 4 days now and it's pretty close to feature complete - yesterday it was able to compile our Game of Life C implementation and the microcode ran on a Tang Nano 9K FPGA board with LCD displaying the cells. I do not think this is an "easier" task.
Last year's state was not good enough for me, but this year's state next year would be great
Interesting, mine freezes the whole laptop:-)
100% agree. A local model might not have the same stretch goal capabilities, but at least I don't have to constantly second guess myself in terms of the inconsistencies I think I'm noticing.
No longer am I constantly made to feel like Ingrid Bergman in that movie with all the lights.
Local AI is the way forward. Using someone else's harness and someone else's model is just borrowing a workflow that's liable and likely to change on a whim.
Assuming you built a top of the line PC a year or two ago or have a huge amount of money to drop on one now