Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today.
Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
(2) Are you saying that you think we're at the limits of computing in general, or that specific technology?
We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.
Leaving aside the discussion on LLMs intelligence vs human intelligence, on a purely energy consumption level we are definitely and without any possible questioning nowhere near that indeed.
Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.
Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.
They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.
Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs.
However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).
Here you have shown yourself that progress slows down and doesnt speed up. 8.9/0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8/8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.
That isn't deceleration, you've just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.
We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.
It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.
iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.
However, the compute situation just skew it even further when it comes to LLMs.
Phones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text
> I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.
The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.
> Those two combined means that unless LLMs reach the state of 'good enough'
Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.
Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.
> Most people are already used to rely on the internet on basically everything
FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.
By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.
No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
That's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.
Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.
Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.
I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).
I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
>As soon as things get in range for motivated amateurs, progress skyrockets.
So true. The reverse is also true - when greedy companies overprice their initial release so that it is out of range of the enthusiastic hobbyist they stall progress and adoption.
This is true for hardware (eg failed Intel Optane, Knights Bridge) and software that does not have a cheap or free basic plan.
1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
Unfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
"It will never be possible to have a computer in your home. You will always have to buy computer time and access via dumb terminals. Technology will never again advance."
it’s equivalent take to laugh at first transformers 9 years ago, because they were shit and hardware requirements were immense.
IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.
The frontier labs will do their best to prevent this from happening. Their financial model won't work if people start running open-weight models on their local hardware.
This is why banning Chinese open-weight AI models is a major policy debate in Washington. The labs can't survive log-term without subsidies, and a ban can act as a subsidy.
I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.
I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.
There are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
>when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.
Agreed. We've just seen DeepSeek post-train their ~300 billion parameter flash model to outperform their 1.6 trillion parameter pro model, in the space of a few months. There would seem to still be quite a few opportunities on the table to bring big model smarts down to the smaller models
A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
> Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]
Unified memory has existed for decades in the PC space, Apple didnt invent it.
And the test you linked to has nothing to do with unified memory, its a web browser benchmark (almost entirely constrained by single threaded CPU performance that Apple better than competitors at).
What might the “parameters/layers to gates” ratio look like? My naive and uninformed guess would be that 1B+ parameter would also need a 1B+ gate FPGA, but according to google they typically range from tens of thousands to several million (which would still be a fraction of a billion).
why would apple
make a chip that could be updated to improve the model when they could just sell you a better chip in the next years device?
on device llm gives apple the new "better camera" "better screen" race they need to keep people coming back for the latest.
for average users everything else is tapped out... screens, cameras wifi... all the core stuff is good enough now its hard to feel/see the difference model year to model year. embedded llm would let them ship something new and the on device ecosystem advantage is huge. especially as the gpt and claudes get ads and enshittified... the apple on device even if its less "capable" would be so compelling.
Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly.
Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.”
Skips the ugly horse race and massive risk while focusing on what they do better than anyone.
And they’re the only ones who have cultivated the trust with normal people to be able to say, “We now have the only AI service that can safely reference your personal info without giving it all to strangers.”
They could possibly end up the second biggest winners in this insane hype cycle… without having had to compete on making a sota model.
Apple has a real knack for not being first mover but being “best” mover. They didn’t invent tablets or mp3 players or thin laptops or ARM laptops, but once they entered those markets they absolutely became the definitional product for the entire category.
Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README.
I am glad it inspired more people to explore area of on-device AI further!
The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?
There are many search APIs available, I like Kagi's.
Microsoft and Amazon both provide web snapshot services that purport to give you a kind of agent-first internet archive. You can approximate something like that using common crawl, but it's a huge amount of data. Downloading the internet is impossible or a bad idea for almost everyone.
This looks useful, you can increase the RAM cache so if you have a Mac with 24-32GB it should speed up a lot and still run models that wouldn’t normally fit. I’m going to run some tests…
I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out.
Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.
> I wonder, is there a way to make the RAM usage tunable?
In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.
It's tunable, --cache-gb N on the CLI. In my sweep the speed barely moved between a 1GB and 6GB cache (43% vs 70% hit rate, same tok/s) because right now the bottleneck is GPU dispatch, not the SSD. so more RAM doesnt buy much yet. once the kernel work lands it should start to matter, so on 32GB I would just set 8 and let it age well. Also the hit rates themselves answer the "can you even know which experts stay hot" question, reuse across tokens is very real.
Achieving high-efficiency model compression to run large models locally on consumer hardware like Macs and iPhones mirrors the architectural goals of decentralized identity.
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.
half an hour to process 10k tokens on an M5 seems... not great
Domestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.
On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.
The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:
1. the organization of the data being sent to the GPU to optimize throughput;
2. the speed at which the GPU can read that data from its VRAM;
3. how many triangles it can process in parallel by using individual compute units.
I suspect that given parallel improvements for neural networks, we'll see similar improvements:
1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;
2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;
3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
If you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.
Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.
Hm, I asked 5.6 Sol to calculate/research and it came up with way less optimistic lifespan than Gemini (that I've asked last year).
It claims that each individual page read induces read disturb across whole block.
And references https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).
Am I the only one that has no flash lifetime anxiety? I still have drives from more than a decade ago that keep on chugging fine.
I remember the time spinning rust was the only option and reliable they weren't. In 30 years of computing I have had more than ten hdds and zero ssds die.
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with <xyz> autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?
When you have Claude Code indepedently author commits and PRs and merge them in, it'll always credit itself as an author. I assume it showing up in the README is a byproduct of the same logic.
no Anthropic involvement, I just used Claude Code heavily while building this and putting that in the README felt more honest than not mentioning it. Now that I think about it may be it shuld be "built with claude code" instead of "built in collaboration...". Changed it.
I know everyone wants to crap all over these setups that are impractical, but this is how progress happens.
People will keep plugging away at this and figure out how to avoid wearing the hard drive, how to make it run faster, custom hardware buses etc.
Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
Haha 1T on $50k might be a bit hopeful, mate, even at FP8. But I too am hopeful.
AMD already demonstrated 1T on strix halo clusters. << $10K at original MSRP.
Problem is prefill on these, right? Initial prompt processing takes forever? I suppose you’re right. Cost is not a thing on its own. It’s a performance-cost frontier and one can do CPU inference in the worst case.
8800 GTX in 2006. Cutting-edge, an insanely powered consumer card for the time. Theoretically around 0.3456 TFLOPS.
1080 GTX in 2016. Cutting-edge, an insanely powerful consumer card for the time. Theoretically around 8.87 to 8.9 TFLOPS.
5090 RTX in 2026. Cutting-edge, an insanely powerful consumer card for today. Theoretically around 104.8 TFLOPS.
In the same timeframe mobile processor CPU's went from 0.001 TFLOPS, to today's Apple's A19 Pro chip which delivers 2.074 TFLOPS.
That's _without_ getting into ASIC's, or purpose-built hardware like Taalas's model on silicon HC1, or generic AI dies like what they're planning with HC2 or Cerebras, which will massively compress the timeline.
Just a note that I think the direction most people are paying attention to is memory bandwidth; thats the real bottleneck and “number go up” but also constraint people are designing around
We will see such power and price now only when AI market crashes or China reaches node parity and goes after market share as currently the way they are buying out most of the latest node production the consumer prices will only be palatable to the very rich or we will need to be happy with older slower nodes
Except the 499$ of a 1080 GTX inflation-adjusted only buys you a 5070 or 5070 Ti even by MSRP.
Sadly while the FLOPS are increasing nicely, total graphics memory is stalled in consumer cards by comparison.
Isn't there such a thing as low hanging fruit?
Aren't we already approaching theoretical physical limits? We're at 2nm
(1) Yes.
(2) Are you saying that you think we're at the limits of computing in general, or that specific technology?
We know, for example, that a human brain level intelligence is possible to run on a human brain. We are nowhere near that. And actually that's not even a physical limit necessarily.
But that is...not a low hanging fruit.
> We are nowhere near that.
Nowhere?
Leaving aside the discussion on LLMs intelligence vs human intelligence, on a purely energy consumption level we are definitely and without any possible questioning nowhere near that indeed.
we are not at 2nm, we just call it that.
Ok, now do memory capacity and bandwidth - the things that actually constraint local LLMs.
8800 GTX in 2006: 768 MB of GDDR3, with 86.4 GB/s of theoretical memory bandwidth.
GTX 1080 in 2016: 8 GB of GDDR5X, with 320 GB/s.
RTX 5090 in 2026: 32 GB of GDDR7, with 1.792 TB/s.
This is fun, what's next?!
PCI 8.0 is breaking 1TB/s, GDDR7 is 1TB/s.
With just the _current_ timeline, things are looking like they'll compress once we get over this initial lump.
Yeah but here you describing the opposite phenomenon. You're saying that the hardware is going to become cheaper and more powerful with the years, to the point a current State of the Art model from today will run on a normal consumer hardware in ten years. What people are trying to do now is the opposite, optimize the software as much as possible so that it does not need the best hardware but the normal one we currently have. As if we were trying to make a current AAA game to run smoothly on the 1080 GTX of your example.
No, both of these things can happen in parallel.
The suggestion is that a 1T model could be made to run on cheap consumer hardware of the future.
Speaking of ASICs - how likely is it that as models get better we'll see someone baking a whole model directly into the silicon? It's like having l0 cache.
You could do it but there would be no point, The only advantage over would be power consumption. And it would be quite expensive.
At the rate models are improving, it would be obsolete in six months.
Power consumption and latency are very important on mobile
They're important everywhere of course, but especially on mobile. If AI researchers figure out how to offload knowledge and expertise from reasoning weights, then a core reasoning ASIC linked to the knowledge would totally rock.
Yes, right now it would be obsolete in six months, but I also must add that this never stopped crypto miners from making new ASICs. However, with how useful Kimi is right now - at some point if someone makes a dedicated hardware board with "good enough" model for daily tasks - that would be a very sought after commodity.
Only 8B currently but it's been done: https://taalas.com/products/
The only _public_ example we know.
This is definitely being done with private models by HFT/quant firms, data processing agencies/orgs (large intelligence agencies, _every_ data analytics org, etc).
Here you have shown yourself that progress slows down and doesnt speed up. 8.9/0.35 = ~25x more performance in 10 years from 2006 to 2016. 104.8/8.9 = ~12x more performance in 10 years from 2016 to 2026. Growth has dropped 50%.
That isn't deceleration, you've just chosen a very selective way to compare. If you use time as a denominator, which is kind of intrinsic when talking about rates of acceleration, you get a very different result. If you graphed .3, 8.9, and 104.4 on the y axis, with years on the x axis, it would be pretty clear that there was in increase in the rate of progress.
We went from adding 8 teraflops in a decade, to adding almost 100 the next decade. If we add "only" 400 more teraflops in the next decade the graph will make that initial growth look flat in comparison, even though your math would show that we are basically stalled out.
It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
>It’s like claiming that a company that goes from making $1 to $1k to $100k to $1mm in a 4 year period has decelerating growth.
Because it is decelerating growth. There is a reason why we use YoY percentage in annual and financial reporting.
We'd bought 4 x $11K Mac Studios at my college and via exo, we had Kimi K2.5 at 30 TPS.
Not too wild an idea!
30 tok/s generation? Nice! What was prefill by the way? Also, these are 256 GB RAM studios? Good timing on those!
Most people are already used to rely on the internet on basically everything. At best, they download a tiny chunk of entertainment from it when they go on a plane, and as soon as they land they immediately abandon that offline chunk.
In addition, LLMs, small or large, are highly parallelizable. This means that running on the same machine/GPUs many requests in parallel is significantly more efficient, and the sum of tok/s will far outweight any single user use case.
Those two combined means that unless LLMs reach the state of 'good enough' (TBD), I fully expect the economics and people's interest to align with 99%+ of LLM usage would be in centralized servers. (not dismissing the fact that there are use cases for local LLMs, and that the progress matters).
I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens. But we could say the same thing about a lot of the computation that iPhones do locally. They could have been much thinner clients, but instead they now have more compute power than desktops had when iPhones launched.
iPhone are in practice more reliant on central servers than ever before. Except for some games, if you take a random person's iPhone it becomes almost useless without internet connection. Which is exactly my point, it's not about compute, it's about people's expecations and priorities. The main driver of the need for higher performance chips on iPhones is its camera and smooth 120fps UI animations, not local processing for apps for almost all people.
However, the compute situation just skew it even further when it comes to LLMs.
Phones are communication devices. So I would always expect connectivity to be central to their use. But that's a different issue than the "inefficiency" of edge computation and whether powerful edge devices can sell (they obviously can).
A lot of the compute that iPhones do either has high bandwidth requirements (photo editing etc) or low latency requirements (games etc). LLMs are an interesting case where unless you're feeding in images, it's just text
> I suspect the economics favor centralized servers, if you only look at the aggregated cost to serve X number of users' tokens.
The economics of real-time, low-latency inference of very large near-SOTA models will heavily favor a centralized setup. But if you can afford to wait for your answer - be it a day, a week, or even more at the extreme low end (or if you just stick to leaner models for your relatively quick replies) the economics start to shift in a very clear way. A slow-going local inference setup relying on cheap SSD offload does not need the high power input of a datacenter rack, and the cooling load is outright trivial - even when working on many requests in parallel, which (in a SSD offload context) is what maximizes throughput even for local inference. These are serious problems for centralized inference that will probably limit the scale at which it can be applied.
I don’t understand the downvotes. It’s perfectly reasonable to disagree, but the argument is laid out clearly. Why not engage with it instead?
This is a nice personal point of view. And you don't care much about privacy or being exposed to the conduct of others completely, do you?
> Those two combined means that unless LLMs reach the state of 'good enough'
Once you are able to not just go on-premise, but on-device, there a boatload of other advantages than efficiency (network independence, privacy, etc.) that make it attractive.
Tiny language models are already plenty "good enough" for a lot of on-device use-cases. E.g. there are a lot of things, that can be reformulated as ranking problems, where you can apply pairwise ranking, which even small language models are good at.
> Most people are already used to rely on the internet on basically everything
FAANG has trained them well. "Normal" people also use chatGPT & co. to ask mundane questions that qwen27b could easily answer (random trivia, writing emails, correcting translations etc), why do these simple requests need to be processed on a B300 on server on the other side of the planet? There is no reason.
And this being hackernews, I would expect people here to go against the grain in this regard...maybe even promote local llms to "normies".
Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?
Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?
Nothing to understand. Straight up hallucination. I could have sworn I read that they used a novel architecture where the model is dense but you could select specific layers or something at inference. reread the announcement: just said MoE. Corrected my brain's weights so thanks.
https://machinelearning.apple.com/research/introducing-third...
So every script kiddie gets its Mythos to hack sides and scammers don’t need AI services anymore.
That time won’t be as much fun as you think
By that time the same Mythos class run on specialized hardware 5 years earlier will have fixed the same bugs that the Mythos class on consumer devices could have exploited.
We'll be fine.
No, CVEs are on average still actively being exploited for far longer than 5 years. The oldest CVEs on CISA’s KEV (Known Exploited Vulnerabilities) catalog are from early to mid 2000s.
That's status right now, and that's because we just got models than can speed up the mitigation or find new CVEs. Plenty of reports of how many of them get patched now, most recently with Chrome.
If that's the natural outcome then the future you're describing is inevitable.
Oh well, maybe the last one to leave the internet can turn the lights off.
It’s not a natural outcome it’s a decision
Agree with this. As soon as things get in range for motivated amateurs, progress skyrockets.
Has also been the case for things like chess computing; a lot of the progress we made over the last decades there (even before involving neural networks!) happened thanks to software improvements because the problem got so accessible, not just faster hardware.
I expect similar trends with AI; I'd expect to get decent, human comparable capability with <200GB/s of memory bandwidth and under 60GB of RAM long term (SSDs with very high read bandwidth looks also promising, but we'll see).
I hope that in a decade or two, training will also be somewhat feasible for "pro-sumers".
> As soon as things get in range for motivated amateurs, progress skyrockets.
That's how the AI boom came around to start with, before CUDA it was an academic topic.
Is this where with in conjunction with Jevon's paradox, we are AI usage to exponentially skyrocket?
>As soon as things get in range for motivated amateurs, progress skyrockets.
So true. The reverse is also true - when greedy companies overprice their initial release so that it is out of range of the enthusiastic hobbyist they stall progress and adoption.
This is true for hardware (eg failed Intel Optane, Knights Bridge) and software that does not have a cheap or free basic plan.
> but this is how progress happens.
This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.
This project is essentially the MoE-of-the day, with some platform-related optimizations.
> Keep going! I personally can't wait for the day when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
That won't happen. Projects like this just give the illusion that that will be possible.
> This is progress in the same way that a man climbing a tree is making progress toward reaching the moon.
That is, an essential step?
There are several aspects to this:
1. Better hardware optimized for matrix operations and other ML/NN specific workflows (NVIDIA hardware, Apple M4/M5 chips, Intel AI Boost, other NPU/TPU chips, etc.).
2. Bigger RAM/VRAM/unified for loading models in memory -- with 3090/4090 you can run 22-24B models in VRAM; with 5090 you can run 30-32B models; with Apple hardware with 128GB unified RAM you can run 70-128B models.
3. Better quantization (smaller, better value representations/lower errors/loss, optimized CUDA/etc. kernels) will lead to larger models being able to be run on the lower VRAM hardware at (with optimized kernels) decent tokens/s.
4. Better optimizations in general taking advantage of the hardware (coupled with the hardware and driver advances) and the NN architectures (NOTE: I saw a huge performance boost a while back after upgrading my llama.cpp version).
5. Better model architectures optimized/designed for running on lower-spec hardware, like Mixture-of-Experts which make it easier to run on a single GPU or distribute the weights more efficiently across multiple GPUs (i.e. you know that specific weights will be used together so you can keep them resident on a specific GPU and not have to swap the weights in/out of the various GPUs or transfer intermediate results of a single layer, only the result from that block/expert).
5. Improved NVME or similar SSD technology which a) has a large amount of storage, and b) has access efficiency on par with RAM/VRAM, especially if/when coupled with efficient DMA access to the GPU/NPU.
Unfortunately you are wrong. Time marches on and progress in silicon does too. You may not like the timeline, but it will happen.
While not exactly an “SSD”, High Bandwidth Flash will make it to market and make big sparse MOE models even more accessible. Everyone knows bandwidth is the game right now and you think hardware vendors are going to sit on their heels and not take part of the market? Absolutely silly to discount progress both on the software and hardware sides.
"It will never be possible to have a computer in your home. You will always have to buy computer time and access via dumb terminals. Technology will never again advance."
it’s equivalent take to laugh at first transformers 9 years ago, because they were shit and hardware requirements were immense.
IMHO it’s a matter of time until we (consumers) will get the hardware (maybe coupled maybe even more novel techniques). Though I expect it will take another 10 years or more.
The frontier labs will do their best to prevent this from happening. Their financial model won't work if people start running open-weight models on their local hardware.
This is why banning Chinese open-weight AI models is a major policy debate in Washington. The labs can't survive log-term without subsidies, and a ban can act as a subsidy.
> That won't happen
I don't understand how anyone with any degree of technological or scientific literacy or knowledge of advances in the past 100-200 years can say something won't happen in the future.
I am absolutely certain it will - the only uncertainty is how long, but given the rate of progress, I'm guessing it won't be far away.
Technically, this can happen. Politically, this won't be allowed to happen - just like digital media ownership never happened.
There are certain physical limits. Calculations need to be done. Either less calculations are necessary for the intelligence, or u accept less intelligence. But there is a limit in what u can do with specific hardware.
> wearing the hard drive
Do reads wear out nvme? Iirc only writes do
only writes and honestly that’s a bit overblown they last a long time
If 1t param model would run from SSDs those will stop costing only $200.
Agree—and I want to see this functionality move to the larger platforms that everyone uses like LM Studio.
llama.cpp has used mmap by default for years
>when a 1t param model runs off a $200 SSD instead of a $50k rack of Nvidia chips.
I am 100% sure this won't happen in 10 years time. At least not on a $200 SSD. But I wouldn't be surprised if it ran on $1K to $2K HBF SSD. It will still be better than a $50K Rack.
I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
It could be on software side too. OpenAI has certainly not plateaued.
The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.
Agreed. We've just seen DeepSeek post-train their ~300 billion parameter flash model to outperform their 1.6 trillion parameter pro model, in the space of a few months. There would seem to still be quite a few opportunities on the table to bring big model smarts down to the smaller models
A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
[1] https://youtu.be/x26A28DoT-w?t=605
> This is why even their single core mobile chips in the budget line Neo
That’s a six core processor. It’s an A18 Pro in the Neo, same chip as on the Iphone 16 Pro
> Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]
Unified memory has existed for decades in the PC space, Apple didnt invent it.
And the test you linked to has nothing to do with unified memory, its a web browser benchmark (almost entirely constrained by single threaded CPU performance that Apple better than competitors at).
I think a major factor is memory bandwidth. Apple has raised it steadily for each M series generation, and that hasn't plateaued.
Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.
I could definitely image Apple embedding a kind of LLM-optimized FPGA: slow to load (update) an LLM, but blazing fast at computing tokens.
Who needs memory when your model is set in silicon ?
You don't an FPGA if you're taping out your own chips. But that is just a MMA accelerator with decent memory bandwidth. No secret sauce here.
I am talking about reconfigurable gates to implement an LLM in silicon, i.e. an FPGA...
What might the “parameters/layers to gates” ratio look like? My naive and uninformed guess would be that 1B+ parameter would also need a 1B+ gate FPGA, but according to google they typically range from tens of thousands to several million (which would still be a fraction of a billion).
why would apple make a chip that could be updated to improve the model when they could just sell you a better chip in the next years device?
on device llm gives apple the new "better camera" "better screen" race they need to keep people coming back for the latest.
for average users everything else is tapped out... screens, cameras wifi... all the core stuff is good enough now its hard to feel/see the difference model year to model year. embedded llm would let them ship something new and the on device ecosystem advantage is huge. especially as the gpt and claudes get ads and enshittified... the apple on device even if its less "capable" would be so compelling.
The innovation curve doesn’t always reward the innovators. It pays to let others test the waters before you.
Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly.
Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.”
Skips the ugly horse race and massive risk while focusing on what they do better than anyone.
And they’re the only ones who have cultivated the trust with normal people to be able to say, “We now have the only AI service that can safely reference your personal info without giving it all to strangers.”
They could possibly end up the second biggest winners in this insane hype cycle… without having had to compete on making a sota model.
Apple has a real knack for not being first mover but being “best” mover. They didn’t invent tablets or mp3 players or thin laptops or ARM laptops, but once they entered those markets they absolutely became the definitional product for the entire category.
We don’t talk about AVP.
Don’t forget AirPods, Apple TV, and Apple Watch. They were not first but they are currently best. The Alexa thing and AVP are notable fails.
Thank you for using TurboFieldfare as a starting point for this project and thank you for mentioning it at the README. I am glad it inspired more people to explore area of on-device AI further!
Is there something that already runs like this on android / linux ( / windows)? (ollama or something?)
Or could this be ported to work on other such platforms?
edit: AI mentions a "BigMoeonEdge" project
The web and connecting to other services is very important for almost all of my use cases. While I believe we are going to get better and faster models, the web index is certainly not downloadable and maintainable for 99.99% of the folks who are able to use local models. Any good solutions exist?
There are many search APIs available, I like Kagi's.
Microsoft and Amazon both provide web snapshot services that purport to give you a kind of agent-first internet archive. You can approximate something like that using common crawl, but it's a huge amount of data. Downloading the internet is impossible or a bad idea for almost everyone.
If you're really into self-hosting I've been experimenting with SearXNG and early signs are promising
This looks useful, you can increase the RAM cache so if you have a Mac with 24-32GB it should speed up a lot and still run models that wouldn’t normally fit. I’m going to run some tests…
I know relatively little about the workings of LLMs, but I keep seeing projects like this that run massive MoE models using very modest amounts of RAM, perhaps excessively so. I wonder, is there a way to make the RAM usage tunable? I have a Macbook with 32 GB of RAM, and it'd be great if I could run the same model but take advantage of the additional RAM to make it run faster.
That's about the turning point for just using typical quants for me. Larger still and you can just do the full model. Smaller to this degree and you need all sorts of extra tricks to get anything.
I guess you have to know which experts to keep “hot” in ram, which you can’t know beforehand, so there wouldn’t be much gain.
I do wonder if there are some experts that are more likely to be hit. So if the normal optimised setup runs in 12GiB an you have 4GiB extra to spare, you could say “promote the most used X experts to this stable (old gen in GC parlance) region and don’t swap it out.
Maybe you could even do something like profiling and remember over multiple sessions (per project/workspace) what the most used agents are and load those up before hand.
> I wonder, is there a way to make the RAM usage tunable?
In LM Studio I can tune it by selecting different quantation of the model, by selecting how many layers of the neural net to be loaded to GPU (rest stays in main mem, evaluated by the CPU), and by adjusting context window.
It's tunable, --cache-gb N on the CLI. In my sweep the speed barely moved between a 1GB and 6GB cache (43% vs 70% hit rate, same tok/s) because right now the bottleneck is GPU dispatch, not the SSD. so more RAM doesnt buy much yet. once the kernel work lands it should start to matter, so on 32GB I would just set 8 and let it age well. Also the hit rates themselves answer the "can you even know which experts stay hot" question, reuse across tokens is very real.
"As far as we know, that is the first time a model of this class has run natively on a phone.".
I feel like I've seen a similar statement on a lot of these streaming weight projects.
400b model on an iPhone: https://x.com/anemll/status/2035901335984611412
Achieving high-efficiency model compression to run large models locally on consumer hardware like Macs and iPhones mirrors the architectural goals of decentralized identity.
Running a 35B on iPhone at 1 tok/s with 2.5 GB RAM… this is the future of on-device inference. Insane work.
> One expectation to set honestly
Hello Claude!
Let me just make sure first instead of guessing
That line was authored by Claude. There you go: https://github.com/leonickson1/Swiftlet/commit/3ac64020eadcb...
How can I do this with, say, Gemma?
Yeah, original TurboFieldfare supports Gemma https://github.com/drumih/turbo-fieldfare
this is cool but like, are we just vibe coding NAND burners at this point? these decode times don't really tell the whole story, because prefill becomes the bottleneck.
half an hour to process 10k tokens on an M5 seems... not great
Not great for coding, or realtime agent interactions. But for background processing tasks overnight? Seems like it’d work pretty well
I'm pretty sure one can rent a GPU for a few minutes with the electricity cost of leaving an M5 overnight.
Domestic electricity is free nowadays, certainly for most of the year, as solar plus battery covers your usage for a tiny percentage of the cost of your house.
Only if you don't count the cost of the equipment and installation.
Sure. One could. But then one wouldn’t be in control of every step of the process.
This is how progress happens, someone gets to 3t/s, the next person gets to6/s and eventually we get to 100t/s.
People like this person are laying the foundations.
There is a limit how much you can squeeze out of given hardware. Betting on it being closer to 100t/s than 6t/s is only that, a bet.
On my 4090 setup I'm getting 86t/s on a 12B Q6_K quantized model running entirely in VRAM.
The current GPUs are optimized for processing huge numbers of triangles per second. There are three things at play here:
1. the organization of the data being sent to the GPU to optimize throughput;
2. the speed at which the GPU can read that data from its VRAM;
3. how many triangles it can process in parallel by using individual compute units.
I suspect that given parallel improvements for neural networks, we'll see similar improvements:
1. optimizing the structure of the weights in the model for efficient access by the CPU/GPU/NPU/TPU;
2. efficient access of data strides (matrix rows) in the memory, e.g. being able to read multiple 2x2 matrix values in one clock cycle, or stepwise pairs of values (a(i,j), b(j,k)) needed for matrix multiplication;
3. parallel compute for matrix and tensor multiplication and other operations needed by neural networks.
If you've got X GB of weights in slow access memory (be it RAM vs VRAM or SSD vs RAM) and Y GB of fast memory then no matter what, if you want to use them you'll need to transfer X-Y GB and will be bound by memory throughput. You can try to reduce number of activated weights, but how much can be gained that way is speculative so far.
SSD NAND reads are nearly infinite. Still makes me uncomfortable, but writing is what kills. There's a reason SSDs are rated by TBW, not TBR.
See also: https://github.com/Helldez/BigMoeOnEdge
At what, 10 tokens per hour? These disk swapping methods all have the same drawbacks - kill your drive early, and slow as hell.
It says very prominently in the post: 4.5-5t/s for 80b on an M5
Isn’t it only writes that kill drives?
Yes for NAND, and I suppose nobody is using mechanical hard drives for this.
I'd like to see someone try it, just to see how incredibly slow and noisy it is.
Yeah, that's why most of these comments seem weird to me.
There's read disturb on SSDs, enough reads will eventually force controller to rewrite the cell and it's neighbours.
Practically if you're not streaming weights 24/7 from a full SSD, then it shouldn't be a problem.
Read disturb ought to be quite rare, especially on a fresh drive that was written only once or a handful of times (WORM-like usage). Practically, it's not likely to be an issue even with very heavy read workloads.
Hm, I asked 5.6 Sol to calculate/research and it came up with way less optimistic lifespan than Gemini (that I've asked last year).
It claims that each individual page read induces read disturb across whole block. And references https://arxiv.org/pdf/2501.02517 that tested Samsung 3D TLC and found ~518K sequential page reads in a block to be ECC threshold (although it's unclear how they got 518K number -- e.g. is it single worst chip they've tried? authors brings up 160 chip sample size later on).
With 7704 pages in a block that's only ~70 sequential block reads till data is lost and to retain data controller would have to refresh block fair bit earlier.. basically it gives modern 3D TLC SSD lifespan measured in months (1TB drive 24/7 sequential reads at 5GB/s).
Am I the only one that has no flash lifetime anxiety? I still have drives from more than a decade ago that keep on chugging fine.
I remember the time spinning rust was the only option and reliable they weren't. In 30 years of computing I have had more than ten hdds and zero ssds die.
I see this at the end of the README
> Swiftlet was built in collaboration with Claude Code.
Did this really happen (some sort of working with Anthropic or Claude Code team) or is it some kind of requirement when you develop some software with Claude Code (I see the other author is: https://github.com/claude), or sort of reuse some of its parts?
Is it like someone saying "built in collaboration with VS Code" or ".. in collaboration with <xyz> autocomplete plugin"?
Or merely a disclaimer about vibe-coding or AI written tool?
It means they used Claude code to write the software.
When you have Claude Code indepedently author commits and PRs and merge them in, it'll always credit itself as an author. I assume it showing up in the README is a byproduct of the same logic.
no Anthropic involvement, I just used Claude Code heavily while building this and putting that in the README felt more honest than not mentioning it. Now that I think about it may be it shuld be "built with claude code" instead of "built in collaboration...". Changed it.