Optimizing Amazon EventBridge Pipes with Apache Kafka at ProSiebenSat.1
ProSiebenSat.1 Media SE is one of the largest German television companies operating in the entertainment sector, offering various products. Joyn, their streaming application used in the DACH regions across multiple platforms. Serving millions of customers during major events, such as popular shows or live sports, is crucial, where customers expect a fast, seamless experience. Slow responses or unavailability leads to frustration, dissatisfaction, and loss of trust.
In this presentation, we will guide you through the journey of optimizing Kafka within our system, starting from its initial status upon takeover. We will tell why we went with Confluent Kafka and not with Amazon MSK. We will look into integrating Kafka with Amazon EventBridge Pipes and share encountered limitations and issues. Next, we will introduce the specific architectural patterns such as the Claim-check and Point-to-point fan-out to enhance resilience and efficiency.
The talk also outlines a wishlist for AWS, highlighting desired improvements in payload size, monitoring capabilities, and processing speed. We will end with lessons learned and strategies for leveraging Kafka more effectively in complex systems.
