A selection algorithm for focused crawlers incorporating semantic metadata

Abstract

The search results offered currently by majority of search portals are horizontal by nature. This denotes that these search engines intend to index as much web pages as possible and present search results based on these web pages. These results often offer generalized results. Focused Crawlers were built to download web pages relevant only to a pre-specified topic. Searching on these kinds of pages is called as Vertical Search, as it attempts to drill down on a single topic, rather than exploring a plethora of other pages on web which are related to search query in one way or another. In this paper, we propose an algorithm which helps a focused crawler decide whether a web page should be downloaded on not. The selection algorithm proposed in this paper makes use of semantic properties of the content to arrive at a decision. © 2013 Springer-Verlag Berlin Heidelberg.