A novel approach for evaluating web crawler performance using content-relevant metrics

Abstract

Most of the search engines will have Web crawler as an important component to index the Web pages to provide more relevant results. Web crawlers are programs used to download and index documents from the Internet. A focused crawler is a specialized crawler which will search for and index the Web page of a particular topic, thus reducing the amount of network traffic and download. In this paper, we present a novel approach for focused Web crawler to download Web pages related to particular topic. This paper also experimented a set of factors to compute the relevancy of Web documents and utilizes the contextual metadata framework (CMF) to summarize the captured relevancy data that can be used to categorize and sort results and in essence improve the quality of the result set presented to the end user. A base comparison has made with a classical crawler, and the appreciable results have been achieved using our approach.