Improving cancer prediction using feature selection in spark environment

Abstract

Cancer prediction from microarray-based gene expression data has been subject to much research in recent years. Because of its vast number of features and relatively smaller sample sizes, feature selection becomes necessary for improving classification performance. Additionally, the characteristics of this malignant condition may often vary, providing a significant amount of data that requires additional time and resources to process. This research work proposes an Apache Spark-based feature selection for microarray cancer classification. The first aim is to select only the optimal and necessary features obtained by the feature selector(information gain [IG] and correlation-based feature selection [CFS]). Secondly, employ a distributed framework and observe the efficiency of the different feature selectors for classification. Finally, we evaluated our approach in terms of accuracy, precision, recall and ROC (AUC) using three classifiers: support vector machine (SVM), naive Bayes (NB), and decision tree (DT). The results reveal that the NB classifier outperformed in all the cases with IG as a feature selector.