Research of efficiency of technologies for data processing in tasks of big data analysis

Abstract

The paper compares the performance of technologies for big data processing applied for anomaly detection in a database of a billing system. Experiments have been conducted to evaluate the performance for processing big data of two technologies such as Hadoop MapReduce and Apache Spark. The datasets and configuration of the applications used in the experiments are described. The experiment results are presented as a description of the relation between calculation time, data size and imbalance in datasets. Based on the obtained results, a decision making system for choosing a technology for processing data depending on the characteristics of the datasets is proposed.