Evaluating Machine Learning Classifiers for Defensive Cyber Operations

Abstract

Today's defensive cyber sensors are dominated by signature-based analytical methods that require continuous maintenance and lack the ability to detect unknown threats. Anomaly detection offers the ability to detect unknown threats, but despite over 15 years of active research, the operationalization of anomaly detection and machine learning for Defensive Cyberspace Operations (DCO) is lagging. This article provides an introduction to machine learning concepts with a focus on the unique challenges to using machine learning for DCO. Traditional machine learning evaluation methods are challenged in favor of a value-focused evaluation method that incorporates evaluator-specific weights for classifier and sensitivity threshold selection specific to the values associated with cyber defense. A comprehensive unknown threat detection experiment is proposed to quantify a classifier's ability to detect previously unseen threats. The proposed experiments and evaluation methods are applied to a Department of Defense (DoD) Cyber Defense Exercise (CDX) dataset to validate the methodology.