Network failure detection and diagnosis by analyzing Syslog and SNS data: Applying big data analysis to network operations

Abstract

We introduce two big data analysis methods for diagnosing the causes of network failures and for detecting network failures early Syslogs contain log data generated by the system. We analyzed syslogs and succeeded in detecting the cause of a network failure by automatically learning over 100 million logs without needing any previous knowledge of log data. Analysis of the data of a social networking service (namely, Twitter) enabled us to detect possible network failures by extracting network-failure related tweets, which account for less than 1% of all tweets, in real time and with high accuracy.