Development of Kafka Messaging System and its Performance Test Framework using Prometheus

Abstract

In today’s era of internet, communication plays a vital role in every aspect of daily activity, as the number of internet users increases an enormous volume of data is generated. This motivates to develop and use a messaging system that collects and analyzes large volume of generated data. Kafka is a distributed messaging system based on publish-subscribe model with high throughput and fault-tolerant mechanism. Around 40% of fortune 500 companies use Kafka messaging system for processing data. Kafka is used to stream real-time data and it is also used to process the data received from IOT sensors. This paper discusses about the procedure for developing and deploying the Kafka messaging system into the working environment based on container technology. The secure communication is enabled by configuring the Kafka with two security mechanisms. The performance of the developed system is evaluated considering QoS parameters for different test scenarios using Prometheus tool.