A Comparative Study of Machine Learning Based Anomaly Detection For IoT Data Using SPARK

Abstract

Internet of Thing (IoT) environment deals with large amounts of data from sensors and other things. The main problem with Internet of Things based applications right now, is security threats with respect to the large amount of data in transit and at rest. Theft or snooping of information or fault in IoT device can cause the data to have some anomalies. We can scan the network to get details of type of packets being transferred, type of protocols being used, source and destination, etc. These data can be analysed to get meaningful inferences and detect intrusion in a network. Most of the work done in this field is single system, single core system based application. This paper focuses on proposing a network anomaly detection mechanism using Machine Learning method, Random Forest in a distributed setup (cluster of machines) using SPARK. Such a mechanism should detect anomalies in large IoT data in an accurate and efficient manner on an IoT dataset called UNSWNB15.