Network Intrusion Detection in Big Dataset Using Spark

Abstract

Nowadays, huge amount of data is flowing every second; hence intrusion detection task became tedious. Hence, Intrusion detection systems require efficient and improved detection mechanism which could detect intrusive activities and serious threat to network security. Nowadays, huge amount of data is flowing every second; hence intrusion detection task became tedious. In our research work, we have proposed a framework in which a feature reduction algorithm is used for reducing the less important features and then applied the supervised data mining techniques on UNSW-NB15network dataset for fast, efficient and accurate detection of intrusion in the Netflow records using Spark. In this paper, we have used two feature reduction algorithms, namely, Canonical Correlation Analysis (CCA) and Linear Discriminant Analysis (LDA) and seven well known classification algorithms. In order to compare the performance of the proposed framework, five performance matrices such as accuracy, Specificity, Kappa, Mean Abs. Error, FPR, Precision, Recall, ROC Area and Training Time are used.