From logs to Stories: Human-centred data mining for cyber threat intelligence

Abstract

An average medium-sized organisation logs approx. 10 to 500 mln events per day on the system. Only less than 5% of threat alerts are being investigated by the specialised staff, leaving the security hole open for potential attacks. Insufficient information in alert message produced in machine-friendly rather than human-friendly format causes cognitive overload on currently limited cybersecurity resources. In this paper, the model that generates the report in natural language by means of applying novel storytelling techniques from security logs is proposed. The solution caters for different levels of reader expertise and preference by providing adjustable templates, filled from both local and global knowledge base. The validation is performed on case study from Security Operations Centre (SOC) at educational institution. The report generated proves superior to existing approach in terms of comprehension (increased cognition) and completeness (enriched context). The evaluation demonstrates power of storytelling in potential threats interpretation in cybersecurity context.