A Critique Empirical Evaluation of Relevance Computation for Focused Web Crawlers

Abstract

Analogous to the spectacular growth of information-superhighway, The Internet, demands for coherent and economical crawling methods are translucent to shoot up. Consequently, many innovativetechniques have been put forth for efficient crawling. Among them the significant one is focused crawlers.The focused crawlers are capable in searching web pages that are suitable for the topics defined in advance.Focused crawlers attract several search engines on the grounds of efficient filtering, reduced memory andtime consumption. This paper furnishes a relevance computation based survey on web crawling. A bunch offifty two focused crawlers from the existing literature survey is categorized to four different classes - classicfocused crawler, semantic focused crawler, learning focused crawler and ontology learning focused crawler.The prerequisite and the mastery of each metric with respect to harvest rate, target recall, precision and F1-score are discussed. Future outlooks, shortcomings and strategies are also suggested