Robust and Scalable Network Monitoring System Using Apache Spark

Abstract

Monitoring and restricting students’ online activities have a lot of possibilities. A robust state-of-the-art network monitoring system and a proxy server architecture can be built to restrict access, monitor activity, reduce latency and increase throughput. In this paper, we have proposed a monitoring system and proxy servers that are built with scalable technologies. A prototype is built with four proxy servers and a central system that includes a Spark cluster and a Kafka cluster hosted on a single node with three Kafka brokers and one Topic with three partitions. The proposed architecture is very user-friendly, with both the API and the Dashboard hosted on the same server using application dispatching.