After spending several weeks working on a ksql solution to filter billions of events and determine their destination topic, I was disappointed to find that it did not live up to my expectations. I had hoped for a more robust product that would align with our needs. Previously, we utilized a similar filter in Spark, incurring traffic costs for both Confluent and AWS. With kSQL, the advantage was that we could avoid paying for AWS traffic. Read more

I’ve been working with Confluent for the last few months. I’ve been mainly configuring the Terraform project and querying the API for some kSQL setup and tuning. When we had ready our sandbox environment, I wanted to test the throughput. If you follow the docs is straightforward, you need to use the serializer from the Confluent library. In fact if you use Spark you can use some nice wrapper as ABRiS. Read more