Distributed search engine system productivity modeling and evaluation

Abstract

This study extends the current productivity models for a typical Web search engine system, which consists of a Web crawling system and an indexing system. Five different design schemata are characterized according to this model and are compared through power consumption, networking cost, system scale, and query efficiency. The half-WAN scheme, which consists of a WAN-based crawling system and a multi-cluster indexing system, is proved to be the best choice for a large-scale highly-efficient Web search engine. © 2012 ISCAS.