The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago.
I don't see how anyone justifies a $200/month claude subscription. Do you want to rent the tools you use to making a living for the rest of your life?
Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.
I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.
It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).
That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.
I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.
If that scales with parameters I don't see why QAT shouldn't be the norm.
(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)
And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.
There is always quality degradation when quantizing, but some models keep surprisingly good performance at aggressive levels of quantization too. DeepSeek v4.1 flash at 2-bit is still a beast for example. In my own experience I would easily trade off accuracy/intelligence for a vast speedup, especially if your setup simply wouldn't run something at 4-bits.
This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.
If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.
With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.
Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.
Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw
Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.
To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175
Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.
So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.
I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.
Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Ironically, local models are much more efficient when running multiple parallel agents, provided that KV storage requirements are not excessive (not an issue with DeepSeek 4.1 Flash). It does not make any individual agent faster, but it does raise overall throughput quite a bit. The limit is then set by compute constraints rather than memory bandwidth, so something like DGX Spark or AMD Strix/Gorgon Halo can be expected to do better than the Apple-based alternative.
Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious.
> I get that you can concievably get 100 agents to do something interesting
Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible.
> Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Huh? Where are you getting this from? Where did I say anything of the sorts?
43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.
I use lemonade llama.cpp with a 2-bit quant of DeepseekV4F 0731 that runs at about 15 tokens a second. I use it as my "smol" model in omp, with default going to OpenCode hosted Deepseek (faster inference).
I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though!
I've recently set up Ollama on my homelab server, which is a Topton N17 micro-ATX motherboard with a Ryzen 7 7840HS and 64 GiB of DDR5 RAM. It's no AMD Strix Halo, but it's a laptop CPU with a fairly beefy GPU and a decent amount of memory bandwidth.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.
Try using llama.cpp directly and tune it for your hardware. I had gemma 26b a3b running at 400-500t/s prefill and 40ish t/s generation speed with a RTX A3000 GPU with only 4.5GB free VRAM. A better model than gemma itself may help you tube the parameters. Gemma wasn't that helpful in that respect.
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.
Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising.
But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.
Always check the label and if pain persists, see your doctor!
The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next
As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.
This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.
(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)
Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
PS: only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point.
Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.
Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.
Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
I wouldn't judge a project by not slamming in 1.7k prs a week. What a nightmare. How many of those prs are pure slop or not even worth looking at? I would wager 95% at a minimum
I'm not saying 1.7K a week, but they aren't keeping on top of the PRs being raised so the number just keeps increasing, unless bots just close them because of inactivity.
At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them).
Scan for exploits, spyware and viruses, actually automatically format the PRs if they don't match and warn the submitter, automatically upmerge main/master into branches under consideration where possible and ensure all new functionality has test cases (not the same as code coverage).
When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional.
If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.
Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!
I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.
The floor for a functional local AI system keeps getting lower, dispite ram and GPU prices continuing to go up. An E5 workstation with lots of threads, $300 of RAM and at least $100 of GPU gets you something that can do >100tok/s+ prompt processing, and ~20tok/s output. Toss a $500 GPU in there, and it goes from barely usable to pleasingly performant. For less than $800 you have something that holds its own with where the SOTA was less than a year ago.
I don't see how anyone justifies a $200/month claude subscription. Do you want to rent the tools you use to making a living for the rest of your life?
Something to keep in mind with aggressive quantisation, is that it's been empirically proven [1] that (1) quantisation has non-linear performance behaviour, and (2) generally the peak size-to-performance ratio occurs at 4-bit; anything lower and you will see significant reduction in performance.
Also, I've been playing with speech models for the past 2 years, and for me anything below 8-bit is a huge drop in quality; so quantisation for LLMs and for other bespoke modality models, is not 1:1.
[1] "Reliability Scaling Laws for Quantized Large Language Models", TUM, https://arxiv.org/pdf/2607.10855
I imagine that it's just the non-linearity of audio makes it easily apparent how much quantization really matters, and that benchmaxxing is so ingrained in LLMs that if you simply run them through benchmarks, they can do well under quantization, when in reality, they have a similar drop off that they do in audio, it just doesn't slam you in the face at every turn as obviously.
I think your leading statement is a bit more authoritative than what the linked paper actually demonstrates. It demonstrates your claimed qualities for specific algorithms, methodologies, and models.
It explicitly says that Quantization-Aware Training is out of scope, may reveal different trends and has potential for extreme quantization. Heavily-quantized model releases today generally use QAT or something similar, so I don't think the paper can be universally applied to modern quants (e.g. [0]).
That said, I don't see any evidence that TFA does any of that so the paper's claims may be quite applicable there.
[0]: https://huggingface.co/vcruz305/DSV4.1-Flash-1017-SAGE-REXL3...
I recently did some toying around with (very) small language models (<200k). I used QAT from the beginning and was surprised that I got byte identical results from the int8 model vs the float32 one.
If that scales with parameters I don't see why QAT shouldn't be the norm.
(As I was only comparing the QAT f32 model vs the quantization its possible that a not QAT f32 would have performed even better)
Do you think QAT isn't the norm yet?
I expected it to be the norm since GPT-OSS
And things like GSQ-RCO optimization from ISTA labs, EXL that you linked. Preserving bits for certain (large) domains like agentic coding. Pruning others. It really works and creates functional models.
There is always quality degradation when quantizing, but some models keep surprisingly good performance at aggressive levels of quantization too. DeepSeek v4.1 flash at 2-bit is still a beast for example. In my own experience I would easily trade off accuracy/intelligence for a vast speedup, especially if your setup simply wouldn't run something at 4-bits.
This is not fully true. 4 over 6 paper(https://arxiv.org/pdf/2512.02010) for example showed that the initial and last layers of transformers that expand into the high-dimensional vocabulary are hurt much more than the center ones. These layers are kept in fp16 and are hurt much more than others by quantization.
If you think of it from an informational perspective, you can imagine that text tokens are much more high-dimensional than a transformed/down-projected variant.
With this reasoning the central layers are mostly building onto the residual stream, and can do fine with 1b and 2b weights.
Afaik this is already a part of regular quants, Q4 just means the average reduction in accuracy is at that level for the whole model, some weights will be fp16, some int8, some int4, some even lower. 4KM is slightly larger than 4_0 for example.
Really? I didn't know, where does it spec this usually?
I have never seen it neither on the link above nor on HF.
Using tooling built into llama.cpp you can quantize a model yourself into a flat q4 bpw
Other techniques likely beyond the scope of that one utility allow you to quantize some weights more and some less. So most model teams put our "average bpw" numbers.
What do you mean never seen?
On HF, click on any Q2K quantized model in GGUF format, it will show the metadata. Look at the expansion of the blocks. You will a mix of quantizations.
To put this on perspective, consider that with 128GB of unified RAM it can run even fast: https://x.com/antirez/status/2119392658661749175 Like, coding-agent acceptably fast. See later in the reply for the prefill at ~300 t/s but can go even faster for larger prefills.
That video seems to show it takes ~4 seconds to process 43 input tokens. Considering the amount of tokens coding-agents chew through, even when just exploring and not even implementing anything, feels far away from "acceptably fast". Say one file is 500 tokens (small file), it'd take ~a minute just to process it?
You mention "See later in the reply for the prefill at ~300 t/s" but I didn't see that in the video nor in any of the Twitter replies, could you link where that is supposed to be happening? Is that warm context perchance?
all my local models do just fine (128GB Strix Halo, 72GB Blackwell) loading data. I'm not sure you were sold the right bill of goods for "agentic" work. I keep these things busy via mind power for a few hours, then get exhausted.
So I implenented a task queue so when I do have all the ideas, I can just set them to work, and "chew" through the work.
I get that you can concievably get 100 agents to do something interesting, but 99% of my work doesn't seem to need to divine naiver stokes.
Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Ironically, local models are much more efficient when running multiple parallel agents, provided that KV storage requirements are not excessive (not an issue with DeepSeek 4.1 Flash). It does not make any individual agent faster, but it does raise overall throughput quite a bit. The limit is then set by compute constraints rather than memory bandwidth, so something like DGX Spark or AMD Strix/Gorgon Halo can be expected to do better than the Apple-based alternative.
> 128GB Strix Halo, 72GB Blackwell
Lol yeah of course, that's not the Apple hardware showcased in the video. Sure, my 6000 Pro also does 11K in prompt processing, but my comment wasn't generally about local models, I thought that was obvious.
> I get that you can concievably get 100 agents to do something interesting
Uuh, OK, cool. Let those vibesloppers do that process if they'd like to, I think most professional software developers already found that process to produce slop and not production-worthy software, so we continue with our process that produce code of quality instead of aiming to produce as much as possible.
> Other than that, you might just have a skill issue if you rely so heavily of spinning up a bunch of agents to tell you what the next TODO item is.
Huh? Where are you getting this from? Where did I say anything of the sorts?
43 tokens is a tiny input context when compared with anything agentic. Prefill on SSD streaming is proportionally much faster with larger contexts, because expert loads can be reused across tokens and the engine can also opt to preload entire model layers into memory, without waiting for the outcome of expert routing.
This seems great. I have 128GB of unified RAM and a AMD Ryzen AI Max+ 395 that I can use but I feel I'm still not using it as well as I could...
The main/best setup I have running so far is qwen flash next with halogen-server.
But if anyone knows if and how one can run even more impressive stuff, please let me know!
I use lemonade llama.cpp with a 2-bit quant of DeepseekV4F 0731 that runs at about 15 tokens a second. I use it as my "smol" model in omp, with default going to OpenCode hosted Deepseek (faster inference).
I haven't done work lately on Dwarf Star. I got it running a few weeks ago and had my agent make a couple of modifications to it to get it running, but it wasn't materially better than my lemonade setup. I might revisit after this post, though!
https://github.com/lemonade-sdk/llamacpp-rocm
I've recently set up Ollama on my homelab server, which is a Topton N17 micro-ATX motherboard with a Ryzen 7 7840HS and 64 GiB of DDR5 RAM. It's no AMD Strix Halo, but it's a laptop CPU with a fairly beefy GPU and a decent amount of memory bandwidth.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.
Try strata. I’m not sure how well it handles igpus but if it does i’m pretty sure that system will go screaming fast
Try using llama.cpp directly and tune it for your hardware. I had gemma 26b a3b running at 400-500t/s prefill and 40ish t/s generation speed with a RTX A3000 GPU with only 4.5GB free VRAM. A better model than gemma itself may help you tube the parameters. Gemma wasn't that helpful in that respect.
The quantization is probably the most important thing to know and its not clear at all from the dense word salad
It looks like 2 bit. I'm really struggling to understand why this got so many upvotes. Colibri and others already covered this ground months ago.
Yeah - at 2bit the accuracy must be pretty poor?
Vibes from localLLAmA suggest a larger model (e.g. DSF) at a low quant still beats a smaller model (e.g. qwen 27B) at a higher quant.
Where that threshold is crossed is up for debate.
At which quantization? That's the only thing I want to know.
EDIT: I think it's 2 bit? If so, would be curious to see the quality of the output. Not great, I think.
Everything lower than 4/5 is really bad
Well it's bad compared to the original model, but is it bad compared to what else you could run in 64 GB?
probably; 32-64 is the range where local models get extraordinarily useful, and in my experience any sub-4 quants get ridiculously incoherent, which is why those models tend to try to 'wow' people with tok/s rather than benchmarks.
haven't tried it yet though.
Qwen3.8-Flash-Next fits in 80GB on strix halo. That's when I started real "vibe" coding with them. Speed is still an issue if I want to move fast and break things, but I've built a little universe for development (I assume many people have also) and it's cruising.
But yeah, 32-64 is the gateway I think. If Qwen4 can make a leap in the flast-next MoE and hit 64 GB on VRAM w/context, that's definitely the cloud killer to me.
I’m wondering myself: Do bigger models suffer less from quantization? Are there papers?
They seem to degrade a lot less.
You don't need papers, good quants are usually published with benchmark data, for example: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-... which maintains parity with FP8 even at 2 bit.
Always check the label and if pain persists, see your doctor!
The ISTA accuracy looks good on that page, but it's only a chosen handful. Instead, look at most of the graphs on Unsloth's Qwen 3.8 Flash Next - https://unsloth.ai/docs/models/qwen3.8-next
As always, check the quant vs KLD graphs (half way down that page).. I found that not all quants are the same - Unsloth's GLM Q4 -> Q3 degrades a lot faster than their Minimax M3.1 Q4 -> Q3 ggufs.
you can get a lot done with models well under that - the google and qwen 20-30B class models are fantastic
This but also the kv quantization shouldn’t be lower than 8 bits.
Next frontier is achieving good output with that kind of quant dimension
This mentality is really pervasive and yet I've found when a lower than 4-bit quant of a given model was all I could use, then using that was often better than using a 4-bit quant of a model that I could. YMMV. It just really made me rethink my stance on the whole thing. Most people likely see something like 2-bit and just feel like the want to puke without ever actually learning whether or not it can handle their tasks and how well relative to other models they run. I've been running Qwen3.8-Flash-Next at Q4, but was crazy surprised at the performance of Unsloth's IQ1_M of DeepSeek v4.1 Flash.
You don’t know what you don’t know
Qwen3.8-Flash-Next at Q2 is often looping itself with repetitions.
There are solutions for that: https://prismml.com/news/bonsai-2-27b
(and before you crap on it, this is not the same as the original Bonsai, this newer one is much better, and upstream support in llama.cpp etc. is being worked on)
Deepseek v4 flash 0731 in a 2-bit quant has been useful for me. Other models don't seem to hold up nearly as well at 2-bit. Most of my other models are 5-6 bit quants because of this.
This 100%
Reddit is flooded with these breathless headlines “xyz tokens/s on my light switch”. It’s always quantizations below 4 bits per weight. Mostly useless.
I’m confused too and the slop readme doesn’t help at all.
> Shrinks DeepSeek-V4-Flash from 76.87 GiB (which crashes in OOM on 64 GB Linux) down to 56.56 GiB
And later
> Standard IQ2_XXS (76.87 GiB, Unpatched)
Except unsloth’s IQ2_XXS GGUF is 91GB, even their smallest IQ1_S is 82.5GB. So I wonder which model/quant the number even refers to.
Also the mentions of “compression” and “MoE pruning” also make it sound lobotomized even more than unsloth’s IQ1 quants.
PS: only now noticed that 76.87GiB is 82.5GB, which happens to be size of the unsloth IQ1_S. The “technical” paper also mentions IQ1_S at some point. Sounds to me like the author’s model or the author mixed IQ2_XXS with IQ1_S at some point.
Also, just noticed that “moe pruning” likely means “the model is supposed to run 6(DS)/8(Qwen) experts per token per layer, but we only run 4 to save on time”.
Either way, the final gguf is 75% of the size of unsloth IQ1_S, which by itself should tell you a lot about its quality; they (somehow; I have no interest digging that info from that slop paper) shrink the (already IQ1) FFN weights, and shrink some attention weights from Q5 to Q2. I’m surprised it was even able to generate the text in the demo.
Interesting. There are quite a few forks of llama.cpp where streaming weights has been successful, or special quants / decompressing before processing.
- Bonsai Qwen 3.8 27B
- Strata
- Pulsar
- This...
I just wish upstream Llama.cpp would accept the PRs faster - they build multiple times per day, but their PR backlog was 1.7K last I checked, which is a ridiculously impractical number to review with a few volunteers.
I watched the Qwen 3.8 Bonsai feature and it took over 3 weeks from creation to be reviewed, and even then it doesn't include the patches for bugs, that they fixed downstream and simply gave up pushing to upstream because the process was so laborious.
So they end up with delayed or missing model support, and half working implementations like Bonsai's Hadamard Folding, and new models lose out on this kind of optimisation.
I suppose this is fine if you intend your project to be the reference project for forks and encourage this kind of specialisation, but I don't think that's intentional.
At least I can use llamaswap to juggle the inference engines and models on the fly, and abstract the fact I need to install 3 or 4 different ones.
I wouldn't judge a project by not slamming in 1.7k prs a week. What a nightmare. How many of those prs are pure slop or not even worth looking at? I would wager 95% at a minimum
I'm not saying 1.7K a week, but they aren't keeping on top of the PRs being raised so the number just keeps increasing, unless bots just close them because of inactivity.
At some point they will have to cut their losses, strip back to a week's worth of PRs and close everything else to get back on top (perhaps this is actually what is happening, it just takes the bots a while to chug through them).
Edit: just checked and it's now 1.9K PRs.
> which is a ridiculously impractical number to review with a few volunteers.
How would that get fixed? I'm skeptical of lowering the standards of review. I suppose if you have some source of trusted volunteers...?
Scan for exploits, spyware and viruses, actually automatically format the PRs if they don't match and warn the submitter, automatically upmerge main/master into branches under consideration where possible and ensure all new functionality has test cases (not the same as code coverage).
When if comes to the human reviewer unless an issue is structural, don't care so much about the formatting and keep to one human approval. For simple changes, like documentation, comments etc, these can be approved by bots themselves as any fault is non functional.
If you consider that if bugs are noticed they will be fixed by other PRs, it makes less sense now to try to fix all issues in each PR, especially as there's no guarantee this would cover all of them anyway with so many ongoing changes.
I wish people would clean up the ai spammed emoji from the github readme. The readability is just horrible.
you're supposed to have your LLM summarize it
Removing the emoji won't help the readability much on this one. The LLM vibes are dialed up to 11, sadly.
I wish people would just write their readmes themselves.
Most people can’t see a difference between scaffolding and an end product, sadly.
How can CPU be fast enough? What does “modchip” refer to?
This is a very impressive achievement in an unexpected direction. I'm really looking forward to trying out qwen on this system.
yeah, would love to see them work some magic on Qwen3.8 Flash Next
Have you tried Strata?
Strata, Project Maya, the FP8 3x speedup on memory-bound workloads on Ampere cards and now llama-mod chip. I’m amazed with all these optimisations and thinking outside the box. Well done, don’t stop!
I very recently spent a lot of time using deepseek v4 flash 0731 before it was superseded by newer more capable things (glm 5.3-flash, v4.1 flash 1017, etc) and I am exceedingly skeptical of anyone that claims to get good QUALITY results out of it at that size. Whatever they're doing to shrink it will make it worse than an unsloth Q2XS quantization of it or equivalent. That size of model needs something like 150GB of RAM to run properly with full unquantized kv cache and not with a ton of training data discarded.
I remain exceedingly skeptical of people who are claiming to run fairly large models in ridiculously tiny sizes. For comparison a 'regular' version of v4 flash 0731 has file sizes more like this: https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF
Even the 97GB Q2XS will show very significant divergence from the full precision thing. Running a lobotomized model just to say you're doing it is really not accomplishing anything special.
Just because you CAN load it, doesn't mean you SHOULD load it. Sparse models n such ... but I digress.
This is quite an obnoxious Readme.md with AI slop. It wraps around llama.cpp but without seeing what platform assumptions it makes its unclear for other readers whether this is worth testing.
Anyone else finding these AI generated pages really hard to read?
Everyone else, yes. This one is particularly mindbendingly bad. It is almost not human-readable without LLM summary.
hm... this is the first time. TIL something from the HN comments more than the actual article. I'm relatively new to the AI space and the idea of quantization is something I'll have to take forward when evaluating future models.
"2-bit quant" is the "in mice" of "look at how big a model I can fit on this hardware".
Prefill of this approach (prompt processing) is far too slow.
The readme is practically unreadable ai slop as well
It is 2 bit? anybody tested this? is any good?
yeah 2 bit quantization, so probably not that great for actual work
all the pieces are coming into place for the Apple M7 chip, 2028 is going to be great for useful local inference
frontier labs are going to need to have some new tricks up their sleeve that are inaccessible by prosumer level compute