An AI platform built from zero and rebuilt after an outage
Amir Salmani,
- 80 people, no platform team
- live in under 50 working days
- 30,000+ images a day
- rebuilt after a datacentre outage
The business
An AI company of about 80 people sold image generation. It had no platform team, nobody doing operations and no written record of what ran where. Engineers waited on tools. The product couldn’t promise paying customers anything about speed.
It also had a date by which the whole thing had to be shown working.
What it needed wasn’t Kubernetes. It needed its people able to work, its product able to take money, and none of that depending on one person’s memory.
The product
In under 50 working days:
- One sign-in for 81 people across 7 applications.
- The old tools moved, not lost. A code host, an issue tracker and a wiki, each on its own forgotten box: 17 repositories, 875 issues and 586 pages carried over.
- An image API the business could sell. 3 lanes, with paying traffic on top. When the system is full, the lower lanes are refused with a clear message. The top lane never is.
- Speed customers notice. A standard image in 3.4 seconds in the first load test. A repeated request in 65 ms.
Production carried more than 30,000 requests a day. The load test put the ceiling at about 230,000.
The tech
31 machines: 3 bare-metal hosts carrying 28 virtual machines, across 2 sites. The main site was a colocation datacentre with no API, where everything is done by hand. The second was a regional cloud holding operations, DNS, mail and backups.
- Kubernetes with RKE2. Monitoring began on its own small K3s cluster and was later folded in.
- Vault for secrets, on bare metal and outside the cluster. Kubernetes needs secrets to start.
- Delivery by Flux. Nobody deploys by hand.
- A private registry, so nothing is pulled from the internet at deploy time.
- Storage on SeaweedFS first, because Ceph was too heavy for 3 nodes, then Rook-Ceph once the cluster grew: 13 disks, 6.3 TiB.
The GPUs. 4 NVIDIA H100s in one machine, joined by NVLink. Between cards I measured 397 GB/s on NVLink and 3.1 GB/s over the network, so splitting them was never an option. An autoscaler runs between 4 and 16 consumers and checks every 10 seconds.
What broke first. A week after launch, 2,000 requests were stuck and errors were climbing. I found 6 bugs. The worst sent a failed job back to the front of the queue, so one bad job held up everyone. I fixed all 6, the backlog drained to zero and the error rate went to 0%. The lanes above came out of that week.
The outage. Months later the main datacentre went dark, and it wasn’t coming back soon. The backups were fine. The restore had never been rehearsed from end to end, and the second site had never been sized to carry everything. I rebuilt the cluster elsewhere, from bare metal, on Talos and Cilium. It took days. No data was lost.
What I’d do differently
- Write the runbook first. I reconstructed the order of recovery during the incident. It should have been written and tested as part of the build.
- Keep the second site warm. A site that holds backups isn’t a site that can take the load.
- Fewer moving parts. Every component I’d added was one more thing to bring back.