You Don’t Need Kafka: Building Event Driven Systems with GenStage and Broadway

Kiunye Araya makes the case that most teams reaching for Kafka actually need backpressure and fault-tolerant concurrency, not a distributed log — and on the BEAM that ships with the runtime. GenStage supplies the primitive (demand-driven stages where a slow consumer asks for N events and never gets flooded), while Broadway adds the parts real pipelines need: processor pools, batched writes, per-message failure isolation, graceful drain on deploy, and pluggable producers. The worked example runs Safaricom Daraja-style mobile-money callbacks through RabbitMQ on a single VPS into a reconciliation table, with Broadway.test_message/2 making handler tests trivial — no embedded broker required. The closing rubric is honest: Kafka earns its keep with multiple independent consumer groups, weeks-long replay windows, and genuinely high throughput, and if you land there later, swapping Broadway's producer over to Kafka is a config change rather than a rewrite.

Visit Original Article →