Can we serve models on GPUs we own?
Yes. I ran image generation on 4 NVIDIA H100s at more than 30,000 requests a day. The GPUs are the simple part. What decides whether it works is the queue in front of them: who waits, who is refused, and what happens when a job fails.
What goes wrong
Not the model. The queue.
On my platform, a week after launch, 2,000 requests were stuck and errors were climbing. A failed job went back to the front of the line, so one bad job held up everything behind it. Nothing was refusing work when the queue was full.
What I’d build
- Priority lanes. Paying traffic never waits behind a batch job.
- Backpressure. Past a set depth, lower lanes are refused with a clear error. The top lane isn’t.
- An autoscaler on the consumers, not only on the pods.
- A cache. A repeated request shouldn’t touch a GPU. Mine came back in 65 ms.
- The cards in one machine when the model spans them. Between cards I measured 397 GB/s on NVLink and 3.1 GB/s over the network.
What it takes
A load test before launch that tells you your real capacity, and a second one after the first incident.
What I haven’t done
My production experience is image generation. I’ve planned large language model serving but not run it at this scale, and I’d tell you where that changes the design.