A survey on various message brokers for real-time big data

Abstract

In the current scenario, processing the huge information is very difficult in the data pipelines. The solution to overcome the above problem is by using message brokers. This will help in collecting and delivering the huge amount of real-time data. In this work, producer client is designed using java language to fetch the information of network elements using Kafka messaging scheme. The consumer application is configured to fetch information from kafka bus. Moreover, performance metrics of kafka bus is also collected. From the results, it is observed that kafka achieved 100% throughput and latency is 1.3 s for fetching 2 lakh messages. From this, it is understood that kafka is very fast, reliable and fault tolerant messaging scheme. Moreover, from the survey it is shown that kafka is the best messaging scheme.