Lino – an intelligent system for detecting malicious web-robots

Abstract

These days various robots are crawling the Internet, they are also called: bots, harvesters or spiders. Popular search engines use a similar technique to index web pages - they have an autonomous agent (called robot or bot) that is in charge of crawling various attributes of web sites. Lately, this crawler technique is exploited by malicious users, for example harvesters, which are used for scraping e-mail addresses from websites in order to build a spam list for spambots. Recently, robots are also misused to buy flight tickets or do fast bids in on-line auction system. In this paper we present an intelligent system called Lino which tries to solve the mentioned problem. Lino is a system that simulates a vulnerable web page and traps web crawlers. We collect various features and perform a feature selection procedure to learn which features mostly contribute to the classification of visitor behaviour. For the classification purpose we use state of the art machine learning methods like Support Vector Machine and decision tree C 4.5.