I did this manually back in Feb 2024[1], but only got "connector A" mapped out and a partially designed SXM2-to-PCIe interposer designed before getting distracted.
During the process, I discovered Facebook's Big Basin JBOG schematics/specs (from Open Compute), which contained a design derived from NVIDIA's DGX-1 reference implementation. Looks like this clears up some gaps in the current reference. It also explains some of the design choices, like the form factor and similar pinout to OpenPOWER.
Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
V100 is a beast if you feed it the right format. A lot of these older cards don’t do well with lower quants. So instead of using Q4, try Q8 or FP16.
If you’re still getting poor performance, point an Astra agent at it and ask it to benchmark and find the pressure points.
Also speculative decoding has changed the game completely. I’ve managed to 2-3x’d the theoretical max for my setup. Apple Silicon is a bit more limited here, and cannot deal with large DFlash block sizes. But AMD and Intel chips are game.
I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
Good to see this re-posted, I posted it two days ago but it sank without a trace so I'm really happy you posted it again. These V100's are a pretty neat GPU and one of very few that is still - somewhat - affordable.
Might be worth depending on the price but GeForce Nvidia GPUs tend to be a better deal than the server class since their EULA doesn’t allow them to be put in datacenters.
Ha, amazing!
I did this manually back in Feb 2024[1], but only got "connector A" mapped out and a partially designed SXM2-to-PCIe interposer designed before getting distracted.
During the process, I discovered Facebook's Big Basin JBOG schematics/specs (from Open Compute), which contained a design derived from NVIDIA's DGX-1 reference implementation. Looks like this clears up some gaps in the current reference. It also explains some of the design choices, like the form factor and similar pinout to OpenPOWER.
https://docs.google.com/spreadsheets/d/1DfH0T9KWWoc2CpKZsRIT...
Nice, I did a similar thing with OCP documents and an SXM2 server schematic set I found, but ultimately lost interest in the project after I rented a server with 8x V100s and realized it was kind of crap for LLM inference.
V100 is a beast if you feed it the right format. A lot of these older cards don’t do well with lower quants. So instead of using Q4, try Q8 or FP16.
If you’re still getting poor performance, point an Astra agent at it and ask it to benchmark and find the pressure points.
Also speculative decoding has changed the game completely. I’ve managed to 2-3x’d the theoretical max for my setup. Apple Silicon is a bit more limited here, and cannot deal with large DFlash block sizes. But AMD and Intel chips are game.
Good luck and godspeed!
I have 4 of these now of the V100S variety. They are aging but solid. If you are not going above 4 cards you probably don’t need SXM2, but the packaging can be easier to deal with. I have PCI variants and network them with 25Gbps and do RDMA.
I have a brain damaged quant of DeepSeek V4.1 (IQ2_XXS) running at about 80-100 t/s and 950 pps. It’s really not that bad, but not as smart as the full model. I have Qwen Flash Next 3.8 on UD4 K XL at about 200 t/s single, 450-550 aggregate t/s and 3-4k pps. Hard to say which is smarter honestly between the two models.
It’s a great model for local LLM. I wrote my own engine and I havent seem any 4 card benchmarks faster than it yet so these cards definitely have more ceiling than most people thought a few months ago. They don’t have a lot of compute but they have just enough to keep their VRAM busy most of the time :)
What rumtime are you using?
Is your engine public?
Those are excellent numbers, especially on QFN. You say the cards are networked, how many cards per node / how many cards per machine?
Good to see this re-posted, I posted it two days ago but it sank without a trace so I'm really happy you posted it again. These V100's are a pretty neat GPU and one of very few that is still - somewhat - affordable.
Is v100 still good enough for local LLM. I found some cheap 16GB variant, but my clanker told me it's too old for modern llm.
Might be worth depending on the price but GeForce Nvidia GPUs tend to be a better deal than the server class since their EULA doesn’t allow them to be put in datacenters.