Robust Analysis of IT Infrastructure's Log Data with BERT Language Model

Abstract

Now-a-days, failure detection and prediction have become a significant research focus on enhancing the reliability and availability of IT infrastructure components. Log analysis is an emerging domain aimed at diminishing downtime caused by IT infrastructure components' failure. However, it can be challenging due to poor log quality and large data sizes. The proposed system automatically classifies logs based on log level and semantic analysis, allowing for a precise understanding of the meaning of log entries. Using the BERT pre-trained model, semantic vectors are generated for various IT infrastructures, such as Server Applications, Cloud Systems, Operating Systems, Supercomputers, and Mobile Systems. These vectors are then used to train machine learning (ML) classifiers for log categorization. The trained models are competent in classifying logs by comprehending the context of different types of logs. Additionally, semantic analysis outperforms sentiment analysis when dealing with unobserved log records. The proposed system significantly reduces engineers' day-to-day error-handling work by automating the log analysis process.