Log File Analysis

Abstract

Log files record invaluable information about the operational details of applications, database management systems, operating systems, and devices. They are autogenerated “diaries” that keep timelines of all data reflecting every event that took place during the operation of the system. Every web site visitor request and the corresponding responses is registered in an access log file, generated by the web server. Access log files keep the entire operational history. Publicly accessible web applications and e-commerce sites that operate 24 h a day, 7 days a week are exposed to the global Internet community. Analyzing the log file is not only crucial for security reasons but also for assessing the community of visitors, gaining insight into their operational habits, knowing their requests, measuring response times, spotting implementation errors, and locating problems of all levels. This chapter starts with a description that shows how to configure and customize a web server, in order to produce a useful access log file, and describes conceptually various software compositions that will constitute contemporary web analytics applications that deal with log files, inventory and customer data, a hybrid application combining log files and tagging system data, near real-time extensions, and social media aware applications that support data streams and provide a more global image of the way a web application is approached by visitors. The impact and the implications of the technology paradigm shift toward rich Internet applications (RIAs) on web analytics applications in Web 3.0 are taken into consideration, and remedies that solve the reduced log file problem are proposed.