Stress-testing mqtt brokers: A comparative analysis of performance measurements

Abstract

Presently, Internet of Things (IoT) protocols are at the heart of Machine-to-Machine (M2M) communication. Irrespective of the radio technologies used for deploying an IoT/M2M network, all independent data generated by IoT devices (sensors and actuators) rely heavily on the special messaging protocols used for M2M communication in IoT applications. As the demand for IoT services is growing, the need for reduced power consumption of IoT devices and services is also growing to ensure a sustainable environment for future generations. The Message-Queuing Telemetry Transport or in short MQTT is a widely used IoT protocol. It is a low-resource-consuming messaging solution based on the publish–subscribe type communication model. This paper aims to assess the performance of several MQTT broker implementations (also known as MQTT servers) using stress testing, and to analyze their relationship with system design. The evaluation of the brokers is performed by a realistic test scenario, and the analysis of the test results is done with three different metrics: CPU usage, latency, and message rate. As the main contribution of this work, we analyzed six MQTT brokers (Mosquitto, Active-MQ, Hivemq, Bevywise, VerneMQ, and EMQ X) in detail, and classified them using their main properties. Our results showed that Mosquitto outperforms the other considered solutions in most metrics; however, ActiveMQ is the best performing one in terms of scalability due to its multi-threaded implementation, while Bevywise has promising results for resource-constrained scenarios.