eSelect: Effective subspace selection for detection of anomalies

Abstract

Anomaly detection is used for many applications such as detection of credit card fraud, medical diagnosis and computer system intrusion detection. Many interesting real-world data sets are high dimensional. Detection of anomalies in such datasets is hampered due to the curse of dimensionality. In such datasets the anomalies are hidden in smaller subspaces of attributes. However the number of subspaces possible from a given attribute set increases in a combinatorial fashion. Consequently an exhaustive search through all possible subspaces for anomalies is not computationally feasible. In this paper we propose a method for exploring the subspaces in a high dimensional data set in an effective and organized way to detect anomalies within them while avoiding an exhaustive search over all possible subspaces. The new method is called eSelect. Through extensive experimentation we compare eSelect to a well-established subspace selection method and demonstrate that our newly proposed method attains marked improvements. © 2014 Springer International Publishing.