Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
I did something like this with Opus 4.5: https://github.com/cpldcpu/smollm.c
It seems implementing transformers is the new "hello world" of AI assisted coding.
Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.
it only cost them $10,000 in Claude API usage. :)
You get that much usage in a monthly $200 subscription.
And a kitten.
So who's paying the cost?
Corporate customers
API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.
People believe in some wild stuff.
Everyone else that doesn't use the entirety of the plan they pay for.
Retail and society as ANT shares get pulled into index and wealth funds
Fpga is software-driven, model can verify its output easily.
Zero surprise it did it
Not really. It's only 1B active, and he was probably using the 5 gb quant. Also he said it was over 200 tokens per second.
But that still not a breakthrough or anything for such a small active set of parameters that is quantized
This is cool
Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...
Good chance I'm not the only one thinking this: share the code
Repetitive "make it faster" prompting is essentially what all these "RSI" claims are
i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.
I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.
(unfortunately for a decent model it will also be price competitive with a big nvidia card)
Now do silicon. ;)
Did you miss this?
https://architectlabs.com/blog/frostbite
They seem to target general-purpose non-homogeneous parallelism, not inference, right?