A new approach to design a domain specific web search crawler using multilevel domain classifier

Abstract

Nowadays information published in the internet has become a common knack for all. As a result volume of information has become huge. To handle that huge volume information, Web researchers are introduced various types of search engines. Efficiently Web-page crawling and resource repository building mechanisms are an important part of a search engine. Currently, Web researchers are already introduced various types of Web search crawler mechanism for the various search engines. In this paper, we have introduced a new design and development mechanism of domain-specific Web search crawler, which uses multilevel domain classifiers and crawls multiple domain related Web-pages, uses parallel crawling, etc. Two domain classifiers used to identify domain-specific Web-pages. These two domain classifiers are used one after the other, i.e., two levels. That's why we are calling this Web search crawler is a multilevel domain-specific Web search crawler. © 2013 Springer-Verlag Berlin Heidelberg.