datadrivenangel 2 hours ago

Does not say what type of model or size they were serving, but one of the traces mentions Maple, which is a 20b model. Still, getting ~70+ tok/s is quite good.

  • datadrivenangel 2 hours ago

    it only cost them $10,000 in Claude API usage. :)

    • hmate9 2 hours ago

      You get that much usage in a monthly $200 subscription.

      • dofm 1 hour ago

        And a kitten.

      • dvh 1 hour ago

        So who's paying the cost?

        • exe34 1 hour ago

          Corporate customers

        • ralph84 56 minutes ago

          API pricing vs. individual plan pricing is just price discrimination. They're both positive gross margin.

        • stronglikedan 49 minutes ago

          Everyone else that doesn't use the entirety of the plan they pay for.

        • nyc_data_geek 45 minutes ago

          Retail and society as ANT shares get pulled into index and wealth funds

  • risyachka 1 hour ago

    Fpga is software-driven, model can verify its output easily.

    Zero surprise it did it

  • ilaksh 20 minutes ago

    Not really. It's only 1B active, and he was probably using the 5 gb quant. Also he said it was over 200 tokens per second.

    But that still not a breakthrough or anything for such a small active set of parameters that is quantized

codekansas 1 hour ago

This is cool

Next step is tuning your post training pipeline to optimize your model for your chosen FPGA architecture...

chalkycrimp 1 hour ago

Good chance I'm not the only one thinking this: share the code

lxe 35 minutes ago

Repetitive "make it faster" prompting is essentially what all these "RSI" claims are

saejox 1 hour ago

i wonder if we can emulate an FPGA on a rtx 3090? then test it on an AWS EC2 F2 few hours a day.

spwa4 49 minutes ago

I must say I do see one application for this, that you can't really match reasonably otherwise: you can bake weights into an FPGA and then run them for 5W at a speed that will be competitive with a big Nvidia card.

(unfortunately for a decent model it will also be price competitive with a big nvidia card)