Analyzing big security logs in cluster with apache spark

Abstract

Cyber security is the major concern in today’s highly networked environment and logging is the primary way of tracking compliance with the security policies. However analyzing the massive amount of logs has become a “Big Data” problem. Apache Spark is one of the latest and most notable incarnation of Data Flow Models in cluster computing. In terms of security log analysis, it provides an exceptional batch or interactive working environment. In this study, Apache Spark along with its distinctive features is briefly introduced, the challenges related to security logs analyzes are discussed and then some of Spark’s security log analyzing capabilities are demonstrated through a problem related to big security logs. Finally, a sample Spark Application is presented that extracts statistics relevant to the problem.