Anomaly Detection in the Web Logs Using Unsupervised Algorithm

Abstract

Web server in the Internet is vulnerable to be attacked. Analyzing on the web logs is one efficient method to figure out intrusion. Using unsupervised algorithm for anomaly detection is suitable for the big data situation. Therefore, the research designs a framework using unsupervised classifiers for anomaly detection in the web log. In this paper, we concentrate on the statistic features and the character features of the web logs. Using the features, we transform the web logs to vectors. We apply a suitable normalized method for our unsupervised classifiers. The principal component analysis (PCA) and the AutoEncoder (AE) are the theoretical basis for the classifiers. As we know, this paper is the first research applying PCA and AE to the web log anomaly detection combining statistic features and character features. In the simulation, we find the statistic features are efficient for the PCA. When we use the AE, character features are better. Compared with other methods, results show that our model performs better.