Comparing alternatives<\/h3>\nActiveMQ Artemis vs. RabbitMQ vs. Redis vs. Kafka<\/h4>\n
In June 2023, computer science researchers at Baylor University measured<\/a> the throughput and latency of these four dissimilar yet popular products. The study utilized the OpenMessaging Benchmark Framework<\/a>.<\/p>\n While in October 2022 Redpanda published a benchmark<\/a> comparing its product to Apache Kafka, an engineer from Confluent aimed to reproduce the results in March\u2013May 2023. During independent and very detailed evaluation<\/a> with 3 brokers on i3en.6xlarge<\/i>, Jack Vanlightly<\/a> found completely different results. The blog series contains a lot of insightful notes on what needs to be done to properly analyze the two products and provides a thoughtful analysis of the results.<\/p>\n GitHub repository<\/a><\/p>\n In March 2023, researchers from the National Taiwan University tested four popular messaging solutions (see the blog post<\/a> and the paper<\/a>). The experiment encompassed two scenarios: one was run on a single machine, and the other used multiple servers connected via Ethernet. For the throughput benchmark, payload sizes varied from 8 bytes to 512 MiB, while latency was tested with the payload fixed at 64 bytes.<\/p>\n GitHub repository<\/a><\/p>\n Researchers from the University of Antwerp ran this comparison<\/a> in September 2022. The benchmark used a fixed RSU, which communicated with a mobile OBU over 4G with a payload size of 256 bytes.<\/p>\n In April 2022, StreamNative\u2014a developer of a cloud-based streaming data platform powered by Apache Pulsar\u2014published another performance benchmark<\/a>. Comparing Pulsar v2.9.1 against Kafka v3.0.0, the tests evaluated Apache Pulsar with and without journaling.<\/p>\n In July\u2013August 2020, Confluent\u2014a company behind a cloud-native data streaming platform\u2014conducted a benchmark<\/a> between Apache Kafka, Apache Pulsar, and RabbitMQ. The tests used three Amazon EC2 i3en.2xlarge instances for the producer, broker, and consumer.<\/p>\n GitHub repository<\/a><\/p>\n Other performance benchmarks that compare Apache Kafka to its alternatives:<\/p>\n In July 2023, Azul conducted a benchmark<\/a> that compared Apache Kafka running on Azul Platform Prime JVM vs. OpenJDK. The tests were conducted with three Kafka broker nodes and one node for Zookeeper. The experiments revealed that disk speed and choice of the filesystem influence Kafka performance significantly.<\/p>\n GitHub repository<\/a><\/p>\n In May 2023, Aiven benchmarked<\/a> Kafka v3.3 on AWS Graviton v2\/v3 processors against Intel- and AMD-powered instances. A 3-node Kafka cluster was run on 6 different architecture instances, evaluating performance, stability, as well as pricing per hour and per 1 billion messages.<\/p>\n In October 2021, the Instaclustr team issued a benchmark<\/a> evaluating Apache Kafka on AWS Graviton2 instances, while exploring the benefits of using gp3 SSDs to boost performance. The study highlights what needs to be done to make this configuration work and how Amazon Corretto Crypto Provider (ACCP) helped during the process.<\/p>\n In July 2021, an architect from Red Hat evaluated<\/a> the performance of Kafka v2.6 with Quarkus v1.11.1. He was using the AMQ Streams v1.6 distribution (Red Hat’s data streaming platform based on Kafka) on OpenShift Container Platform v4.5 (built upon Kubernetes). Rather than benchmarking the maximum performance, Berker Agir aimed to demonstrate how different configurations impact producer message throughput.<\/p>\n GitHub repository<\/a><\/p>\n More performance benchmarks like that were issued by infrastructure vendors:<\/p>\n Benchmarks indicate that the performance of Apache Kafka may depend on the scenario, configuration, and the platform it is running. Thorough technology benchmarking<\/a> can reveal unexpected behavior or give other insights vital to understanding how the evaluated tool fits the specific requirements of your project.<\/p>\n In addition to 20+ studies above, the performance reports below can also be helpful. Though they do not compare Kafka to other platforms, they reveal how performance can change under various requirements, implementations, and optimizations.<\/p>\n This collection of performance comparisons may not be exhaustive\u2014new evaluations are being released almost each month. Therefore, if you know any other Apache Kafka benchmarks we missed\u2014feel free to add them in the comments.<\/p>\n What is Apache Kafka?<\/strong><\/p>\n <\/div>\n Apache Kafka is a robust, open-source stream-processing platform that has revolutionized how companies process and analyze real-time data. Kafka uses a binary TCP-based protocol and is designed to handle data streams from multiple sources, delivering messages to multiple consumers with high performance and in real-time. Kafka was originally developed at LinkedIn in 2010 as a solution to improve the company's data pipelines. In 2011, Kafka was open-sourced as part of the Apache Software Foundation, making it available to a broader community. How does Apache Kafka differ from Pulsar and RabbitMQ?<\/strong><\/p>\n <\/div>\n Apache Kafka's architecture<\/a> is fundamentally designed as a distributed commit log. At its core, it uses a broker system where each broker is responsible for storing data and serving clients. Kafka organizes messages into topics, and each topic is divided into partitions to ensure parallel processing and scalability. Producers publish messages to topics, and consumers subscribe to topics and process the feed of messages. Kafka maintains high performance and durability by replicating the partitions across multiple brokers and uses Zookeeper for broker coordination and maintaining the cluster state. Kafka's architecture excels in scenarios requiring high throughput and durability, making it ideal for building real-time streaming data pipelines<\/a> and applications. How can the performance of Apache Kafka be optimized?<\/strong><\/p>\n <\/div>\n To improve Apache Kafka performance, consider balancing the number of partitions<\/a>, optimal replication factors<\/a>, adjusting broker topic<\/a> configurations, optimizing producer settings like batch size and compression, etc. It is also important to fine-tune consumer fetch sizes and manage hardware resources efficiently. How much throughput can Apache Kafka handle?<\/strong><\/p>\n <\/div>\n The performance of Apache Kafka can be significantly high, but it's not a fixed number. The throughput depends on various factors\u2014including hardware configuration, network setup, configuration, topic partitioning, message size, and the specific workload being run. Generally, Kafka is known for its ability to handle massive load scenarios. How can one measure the performance of Apache Kafka?<\/strong><\/p>\n <\/div>\n If you are looking to conduct your own Apache Kafka performance tests, there are several tools that can be used for the process. For instance, the OpenMessaging<\/a> benchmark by Confluent is one of the most popular tools used within the industry. Another application that can be used is JMeter\u2014see the docs<\/a> and read this guide<\/a> from Blazemeter to learn how to test Apache Kafka with this product. Explore evaluations of Kafka\u2019s performance on various cloud platforms, as well as benchmarks against other messaging tools.<\/p>\n","protected":false},"author":7,"featured_media":1591,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[11],"tags":[10],"class_list":["post-1495","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-article","tag-the-internet-of-things","entry"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/posts\/1495","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/comments?post=1495"}],"version-history":[{"count":76,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/posts\/1495\/revisions"}],"predecessor-version":[{"id":1509,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/posts\/1495\/revisions\/1509"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/media\/1591"}],"wp:attachment":[{"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/media?parent=1495"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/categories?post=1495"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.altoroslabs.com\/blog\/wp-json\/wp\/v2\/tags?post=1495"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2023\/07\/activemq-artemis-vs-rabbitmq-vs-redis-vs-apache-kafka-throughput-and-latency-test-results.png\")
<\/a>Throughput and median latency (image credit<\/a>)<\/small><\/center><\/p>\nApache Kafka vs. Redpanda<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2023\/12\/apache-kafka-vs-redpanda-performance-benchmark-jack-vanlightly-2023.png\")
<\/a>Apache Kafka (AK) vs. Redpanda (RP) latencies (image credit<\/a>)<\/small><\/center><\/p>\nMosquitto vs. Zenoh vs. Kafka vs. DDS<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2023\/12\/Apache-Kafka-DDS-MQTT-Zenoh-multiple-machine-benchmark-1024x576.png\")
<\/a>Kafka, MQTT, DDS, and Zenoh: multiple machine throughput (image credit<\/a>)<\/small><\/center><\/p>\nActiveMQ vs. Kafka vs. Zenoh vs. Pulsar<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2023\/07\/activemq-vs-kafka-java-vs-kafka-python-vs-zenoh-pulsar-latency-and-resource-utilization-benchmark-results.png\")
<\/a>Latency and resource utilization results (image credit<\/a>)<\/small><\/center><\/p>\nPulsar v2.9.1 vs. Kafka v3.0.0<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2024\/01\/apache-kafka-pulsar-performance-benchmark-2022-throughput-1024x731.png\")
<\/a>Apache Kafka vs. Apache Pulsar with\/without journaling (image credit<\/a>)<\/small><\/center><\/p>\nRabbitMQ vs. Kafka vs. Pulsar<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2023\/12\/Apache-Kafka-Pulsar-RabbitMQ-benchmark-1024x403.png\")
<\/a>Kafka, Pulsar, and RabbitMQ benchmark results (image credit<\/a>)<\/small><\/center><\/p>\nKafka compared to 10+ other systems<\/h4>\n
\n
Performance on different platforms<\/h3>\n
Apache Kafka on Azul Platform Prime vs. OpenJDK<\/h4>\n
![\"\"](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2024\/01\/Apache-Kafka-Azul-OpenJDK-benchmark-throughput-1024x576.png\")
<\/a>Kafka throughput on Azul Platform Prime and OpenJDK (image credit<\/a>)<\/small><\/center><\/p>\nApache Kafka on AWS Graviton vs. Intel vs. AMD<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2024\/01\/apache-kafka-performance-benchmark-on-aws-graviton-intel-and-amd-2023-1024x461.png\")
<\/a>Apache Kafka performance on AWS Graviton 2\/3 vs. Intel vs. AMD (image credit<\/a>)<\/small><\/center><\/p>\nApache Kafka on AWS Graviton with gp3<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2024\/01\/apache-kafka-performance-benchmark-2021-aws-graviton-gp3-throughput-r6g.2xlarge-1024x587.png\")
<\/a>Apache Kafka performance on AWS Graviton2 with gp3, r6g.2xlarge (image credit<\/a>)<\/small><\/center><\/p>\nApache Kafka v2.6 on OpenShift<\/h4>\n
![](\"https:\/\/www.altoroslabs.com\/blog\/wp-content\/uploads\/2024\/01\/apache-kafka-performance-throughput-2021-quarkus-openshift-container-platform-red-hat-amq-streams-1024x657.png\")
<\/a>Configurations influence Apache Kafka performance (image credit<\/a>)<\/small><\/center><\/p>\nKafka on Kubernetes, OpenShift\/IBM, NetApp, etc.<\/h4>\n
\n
Evaluating configurations\/deployments<\/h3>\n
\n
Frequently asked questions (FAQ)<\/h3>
\r\n
\r\nIn 2012, Kafka became a top-level Apache project<\/a>, signifying its maturity and the community's confidence in its potential and stability. This milestone was crucial in cementing Kafka's reputation as a reliable and scalable choice for handling real-time data streams. The introduction of the Kafka Streams API<\/a> in 2016 marked its evolution from a messaging queue to a full-fledged stream processing framework, enabling users to process their data directly within Kafka. This shift expanded Kafka's use cases beyond just messaging and into the realm of complex event processing and real-time analytics.
\r\n
\r\nOne of Kafka's most influential milestones came in 2017 with the introduction of exactly-once semantics<\/a>. This made stream processing more reliable by ensuring that events are neither lost nor duplicated.<\/p>\n <\/div>\n <\/div>\n <\/li>
\r\n
\r\nApache Pulsar is recognized for its two-layered architecture<\/a>, separating the serving layer (brokers) from the storage layer (BookKeeper). This distinct architecture allows for independent scaling of serving and storage resources, providing flexibility, and efficiency in resource utilization. Pulsar's brokers handle client connections and implement various messaging protocols, while the BookKeeper handles durable storage of messages.\u00a0Similar to Kafka, Pulsar organizes messages into topics and partitions for scalability. However, Pulsar enhances its architecture with features like multi-tenancy and geo-replication out of the box. This allows for effective resource isolation and data synchronization across different clusters.\u00a0Pulsar's architecture is often chosen for its cloud-native design<\/a>, supporting real-time analytics and data integration in a multi-tenant environment.
\r\n
\r\nRabbitMQ follows a more traditional message broker architecture with a focus on flexibility and reliability. It manages and routes messages through exchanges based on rules defined in bindings. Messages are then stored in queues until consumed by applications. RabbitMQ's architecture is focused on ensuring message delivery and providing various routing options. This makes it versatile for different messaging patterns. It supports multiple messaging protocols<\/a>, such as AMQP, MQTT, and STOMP, offering a broad range of compatibility with different systems.\u00a0RabbitMQ's architecture shines in scenarios<\/a> that require complex routing, immediate message delivery, and a guarantee that messages are safely transferred.<\/p>\n <\/div>\n <\/div>\n <\/li>
\r\n
\r\nIncreasing the number of partitions enables more parallelism, thus also improving throughput. However, too many partitions can, vice versa, increase overhead and decrease performance.
\r\n
\r\nA guide<\/a> by Microsoft details how to optimize Kafka on Azure by adjusting producer, broker, and consumer settings. Another tutorial<\/a> (by HUMAN Security, Oct 2023) focuses on fine-tuning Kafka by changing compression types.
\r\n
\r\nOther factors can also increase Kafka's throughput. SSDs, for example, can improve performance over HDDs due to their faster read\/write speeds.<\/p>\n <\/div>\n <\/div>\n <\/li>
\r\n
\r\nBack in 2014, the LinkedIn Engineering blog revealed<\/a> that Apache Kafka can handle more than 2 million records per second on three commodity machines. In 2022, this figure was eventually confirmed<\/a> by Honeycomb for their real-life deployment, which uses the Confluent platform for the purpose.<\/p>\n <\/div>\n <\/div>\n <\/li>
\r\n
\r\nDeveloped by Grafana Labs, k6<\/a> is one more open-source load testing tool. This blog post<\/a> details how to monitor Kafka producers and consumers. Another guide<\/a> demonstrates how to determine the maximum load with Gatling.<\/p>\n <\/div>\n <\/div>\n <\/li><\/ul>\nFurther reading<\/h3>\n
\n
\n<\/a><\/li>\n<\/ul>\n