Multi-Engine Search and Comparison Using the MetaCrawler

Abstract

Standard Web search services, though useful, are far from ideal. There are over a dozen different search services currently in existence, each with a unique interface and a database covering a different portion ofthe Web. As a result, users areforced to repeatedly try and retry their queries across different services. Furthermore, the services return many responses that are irrelevant, outdated, or unavailable, forcing sents the the MetaCrawler, user to manually a fielded sift through Web service the responses that represents searching the for next useful level up information. in the information Thispaper “food prechain.” The MetaCrawler provides a single, central interface for Web document searching. Upon receiving a query, the MetaCrawler posts the query to multiple search services in parallel, collates the returned references, and loads those references to verify their existence and to ensure that they contain relevant information. The MetaCrawler is sufficiently lightweight to reside on a user’s machine, which facilitates serves as a customization, toolfor comparison privacy, of sophisticated diverse search filtering services. of Using references, the MetaCrawler and more. The ’s data, MetaCrawler we present also a “Consumer Reports” evaluation of six Web search services: Galaxy [5], InfoSeek [1], Lycos [15], Open Text [20], Webcrawler [22], and Yahoo [9]. In addition, we also report on the most commonly submitted queries to the MetaCrawler.