Malcrawler: A crawler for seeking and crawling malicious websites

Abstract

Over the years, internet has become the major source of security threat to computer systems. With the number of people browsing internet increasing exponentially in the last couple of years, browser based attacks have become the preferred means of infecting a computer system. These browser based attacks, known as ‘Drive-by Download’ attacks, inject malicious Java-Script from the server hosting the malicious web application to the browser. Since, the numbers of malicious websites launching such attacks have increased in the past few years; it has become critical to detect them. Typically, search for malicious web pages involves three steps-crawling URLs on the internet, using fast analysis filters to reject benign pages, and then running complex but slow detailed analysis (using Honey Clients) on the filtered list. While effective, these techniques consume substantial time and computing resources. This limitation can be overcome by designing a crawler which can seek more malicious sites than benign sites, thus, increasing the “toxicity” of the URLs collected in the first step. In this paper, we propose a focused web crawler, named “Mal-Crawler”, which has been designed to crawl and search malicious websites efficiently. This crawler, when compared to a generic crawler, will not only seek more malicious sites than benign sites, but will also handle cloaking, entanglement and AJAX content in malicious sites. MalCrawler, designed, developed and tested, as part of the scope of this paper, proved to be more efficient than generic crawlers.