When a tier-1 financial institution asked us to rebuild their transaction processing pipeline, the requirements were straightforward: 2.8 billion events per day with exactly-once semantics, sub-100ms latency, and zero data loss.
Kafka as the backbone for durability guarantees. Flink for stream processing with exactly-once semantics. The combination gives us log-based throughput with true stream engine processing power.
The hardest problems weren't technical — they were operational. Schema evolution across 400+ event types, managing consumer lag during deployments, debugging distributed transactions spanning 12 microservices. We built the tooling that didn't exist.