A fleet pipeline with 700,603 checkpoints and none failed
Amir Salmani,
- 700,603 checkpoints, 0 failed
- 19 machines down to 8
- the open question was commercial
The business
A fleet company sells tracking: where each vehicle is and what its sensors read. Its customers pay for that being right. If readings go missing, the product is wrong and nobody can say when.
Later the question changed. The company wanted to grow a long way past its current fleet, and wanted to know what that would cost.
The product
- Every reading from every device, in the database, on time.
- More than one customer on the same system, each seeing only its own fleet.
- A status brief for management. I wrote it for people who don’t read architecture diagrams.
That brief’s main point: the open question isn’t technical, it’s commercial. The system kept every reading forever. A year of full detail, or 90 days of detail plus summaries, changes the hardware bill by a large multiple. Nobody can size the system until someone decides what the data is worth, and that isn’t an engineer’s decision.
The tech
A few hundred devices, each reporting about every 2 minutes. Devices publish over MQTT. A Flink job bridges MQTT into Kafka. Flink processes the stream and writes to PostgreSQL. A Django API serves it.
The record. The bridge job started on 5 August 2025. When I measured it in April 2026 it had completed 700,603 checkpoints, one every 30 seconds, and failed none. Consumer lag was 0.
What was weak. Kafka ran as a single node and the whole pipeline sat on one host. The checkpoint record says the software held. It says nothing about a dead host.
The rebuild. In September 2026 the platform moved onto new machines and the old system was switched off. The fleet came across without losing data, and it runs on 8 machines where there had been 19. The move turned up things that had been wrong and invisible for months: a database with no working backup, and a service exposed to about 9,000 attacks a day.