A dynamic ensemble learning framework for data stream analysis and real-time threat detection

Abstract

Security incident tracking systems receive a continuous, unlimited inflow of observations, where in the typical case the most recent ones are the most important. These data flows and characterized by high volatility. Their characteristics can change drastically over time in an unpredictable way, differentiating their typical normal behavior. In most cases it is not possible to store all of the historical samples, since their volume is unlimited. This fact requires the extraction of real-time knowledge over a subset of the flow, which contains a small but recent percentage of all observations. This creates serious objections to the accuracy and reliability of the employed classifiers. The research described herein, uses a Dynamic Ensemble Learning (DYENL) approach for Data Stream Analysis (DELDaStrA) which is employed in RealTime Threat Detection systems. More specifically, it proposes a DYENL model that uses the “Kappa” architecture to perform analysis of data flows. The DELDaStrA is based on the hybrid combination of k Nearest Neighbor (kNN) Classifiers, with Adaptive Random Forest (ARF) and Primal Estimated SubGradient Solver for Support Vector Machines (SVM) (SPegasos). In fact, it performs a dynamic extraction of the weighted average of the three results, to maximize the classification accuracy.