Combating threat-alert fatigue with online anomaly detection using isolation forest

Abstract

The threat-alert fatigue problem, which is the inability of security operators to genuinely investigate each alert coming from network-based intrusion detection systems, causes many unexplored alerts and hence a deterioration of the quality of service. Motivated by this pressing need to reduce the number of threat-alerts presented to security operators for manual investigation, we propose a scheme that can triage alerts of significance from massive threat-alert logs. Thanks to the fully unsupervised nature of the adopted isolation forest method, the proposed scheme does not require any prior labeling information and thus is readily adaptable for most enterprise environments. Moreover, by taking advantage of the temporal information in the alerts, it can be used in an online mode that takes in the most recent information from past alerts and predicts the incoming ones. We evaluated the performance of our scheme using a 10-month dataset consisting of more than half a million alerts collected in a real-world enterprise environment and found that it could screen out 87.41% of the alerts without missing any single significant ones. This study demonstrates the efficacy of unsupervised learning in screening minor threat-alerts and is expected to shed light on the threat-alert fatigue problem.