Show HN: How We Run 60 Hugging Face Models on 2 GPUs
Most open-source LLM deployments assume one model per GPU. That works if traffic is steady. In practice, many workloads are long-tail or intermittent, which means GPUs sit idle most of the time.
We experimented with a different approach.
Instead of pinning one model to one GPU, we: •Stage model weights on fast local disk •Load models into GPU memory only when requested •Keep a small working set resident •Evict inactive models aggressively •Route everything through a single OpenAI-compatible endpoint
In our recent test setup (2×A6000, 48GB each), we made ~60 Hugging Face text models available for activation. Only a few are resident in VRAM at any given time; the rest are restored when needed.
Cold starts still exist. Larger models take seconds to restore. But by avoiding warm pools and dedicated GPUs per model, overall utilization improves significantly for light workloads.
Short demo here:https://m.youtube.com/watch?v=IL7mBoRLHZk
Live demo to play with: https://inferx.net:8443/demo/
If anyone here is running multi-model inference and wants to benchmark this approach with their own models, I’m happy to provide temporary access for testing.
Ollama does this too
Do you have something people can actually try? If not, Show HN is not appropriate, please see the first sentence here
https://news.ycombinator.com/showhn.html
Ollama is great for local workflows. What we’re focused on is multi-tenant, high-throughput serving where dozens of models share the same GPUs and scale to zero without keeping them resident.
You’re right that HN expects something runnable. We’re spinning up a public endpoint so people can test with their own models directly instead of requesting access. I’ll share it shortly. Thank you for the suggestion.
How many people need to host models like this? I'm having trouble seeing why I would need this muti-tenant model stuff if I'm building a consumer or b2b app
In other words, how many middlemen do you think you TAM is?
You go on to say this is great for light workloads, because obviously at scale we run models very differently.
So who is this for in the end?
Good question.
This isn’t for single-model apps running steady traffic at high utilization. If you’re saturating GPUs 24/7, you’ll architect very differently.
This is for teams that…
• Serve many models with uneven traffic • Run per-customer fine-tunes • Offer model marketplaces • Do evaluation / experimentation at scale • Have spiky workloads • Don’t want idle GPU burn between requests
A lot of SaaS AI products fall into that category. They aren’t OpenAI-scale. They’re running dozens of models with unpredictable demand.
Lambda exists because not every workload is steady state. Same idea here.
> A lot of SaaS AI products fall into that category. ... They’re running dozens of models with unpredictable demand.
How do you know this? What are the numbers like?
> Lambda exists because not every workload is steady state
Vertex AI has all these models via API or hosting the same way. Same features already available with my current cloud provider. (traffic scaling, fine-tunes,all of the frontier and leading oss models)
Vertex is great control plane. We’re not replacing them.
What we focus on is the runtime layer underneath. You can run us behind Cloud Run or inside your existing GCP setup. The difference is at the GPU utilization level when you’re serving many models with uneven demand.
If your workload is steady and high volume on a small set of models, the standard cloud stack works well. If you’re juggling dozens of models with spiky traffic, the economics start to look very different.
As an example, we’re currently being tested inside GCP environments. Some teams are experimenting with running us behind their existing Google Cloud setup rather than replacing it. The idea isn’t to swap out Cloud Run or Vertex, but to improve the runtime efficiency underneath when serving multiple models with uneven demand.
Vertex AI is far more than a "control plane"
I don't see anything you do that they don't already do for me. I suggest you do a deep dive on their offering as there seem to be gaps in your understanding of what features they have
> economics start to look very different
You need to put numbers to this, comparing against API calls at per-token pricing is a required comparison imo, because that is the more popular alternative to model hot-swapping for spikey or heterogeneous workloads
You can try it here: https://inferx.net:8443/demo/
There is nothing there to "try", it's some very basic html displaying some information that doesn't mean anything to me. Looks like a status page, not a platform
Really, it looks like someone who is new to startups / b2b copy, welcome to first contact with users. Time to iterate or pivot
I would focus on design, aesthetics, and copy. Don't put any more effort into building until you have a message that resonates
Basic html? The core of what we built is at the runtime layer. We’re capturing CUDA graphs and restoring model state directly at the GPU execution level rather than just snapshotting containers. That’s what enables fast restores and higher utilization across multiple models.
If that’s not a problem space you care about, that’s totally fair. But for teams juggling many models with uneven traffic, that’s where the economics start to matter.
Also, For what it’s worth, this can be deployed both on-prem and in the cloud. Different teams have different constraints, so we’re trying to stay flexible on that.
Happy to dig deeper and show how exactly it works under the hood. For context, here’s the main site where the architecture and deployment options are explained: https://inferx.net/
I don't personally have this problem. One of my clients does, so my questions are ones I'd expect the CTO to ask you in a sales call. They already have an in-house system and I suspect would not replace it with anything other an open source option or hyperscaler option.
Are you going to make this open source? That's the modus operandi around Ai and gaining adoption for those outside Big Ai (where branding is already strong)
It’s an open-core model. The control plane is already open source and can be deployed fairly easily. We’re not trying to replace in-house systems or hyperscalers. This can run on Kubernetes and integrate into existing infrastructure. The runtime layer is where we’re focusing the differentiation.
It’s an open-core model. The control plane is already open source and can be deployed fairly easily. We’re not trying to replace in-house systems or hyperscalers. This can run on Kubernetes and integrate into existing infrastructure. The runtime layer is where we’re focusing the differentiation.
I'm referring to the "demo" and inappropriateness of the Show HN prefix
there is nothing to try or play with, it's just content
The demo is live. It’s meant to show how snapshot restore works inside a multi-tenant runtime, not just a prompt playground. You can interact with the deployed models and observe how state is restored and managed across them. The focus is on the runtime behavior rather than a chat UI.
Please read the first sentence of the Show HN guidelines, "show" is more specific in this context. This should be a regular submission type
https://news.ycombinator.com/showhn.html
Fair point. I’ll repost as a regular submission instead of Show HN. The goal was to demonstrate the runtime behavior behind multi-model serving rather than a polished end-user app. Appreciate the clarification.
Happy to engage with you if Hulu have any additional questions. Thanks for all the feedback.that was helpful