This post will highlight some practical aspects of using kafka transactions. I will not dive deep into theory on how Kafka transactions are implemented. Instead, let’s discuss main “gotchas” someone could encounter when trying to use Kafka transactions in production first time.
This article is inspired by Monty-Hall problem/paradox. Now I found this paradox disturbing initially (this is why it’s paradox, wink…), and decided to quickly model it to validate for myself. That brought idea of writing up all I know about Java Random API in this article.
Most developers I talk to about Kafka agree on a catchphrase “Kafka is designed for throughput”. That’s fair and you can find plenty of benchmarks that show 700k/s throughput for single producer without replication. But does that mean we should discard Kafka when talking about low latency messaging?
After quick googling I found this outdated SO question. That question is very old and includes very old version of Kafka (0.7.2 whereas current version is 2.0.0) and also this article states that very decent latencies (2-3ms 99 percentile) are achievable. Having such controversial info is not enough to make final decision so I decided to create a little benchmark myself, to finally conclude whether Kafka is good for low-latency applications.
Many times I find an ExecutorService with 400 threads in Java code and I always got that’s-not-ok feeling of that. This is usually introduced as necessary evil to retain application responsiveness while being blocked by downstream. However there is another way of achieving responsiveness, which is couroutines (alternatively known also as green threads and non-blocking threads in another languages). Java doesn’t support coroutines at language level, so for this example we will use almost exclusive Java implementation of coroutines - Quasar.