Exploiting hybrid parallelism in web search engines

Abstract

With the emergence of multi-core CPU (or Chip-level MultiProcessor -CMP-), it is essential to develop techniques that capitalize on CMP's advantages to speed up very demanding applications of parallel computing such as Web search engines. In particular, for this application and given the huge amount of computational resources deployed at data centers, it is of paramount importance to come out with strategies able to get the best performance from hardware. This is specially critical when we consider how we organize hardware to cope with sustained periods of very high traffic of user queries. In this paper, we propose an hybrid technique based on MPI and OpenMP which has been devised to take advantage of the multithreading facilities provided by CMP nodes for search engines under high query traffic. © 2008 Springer-Verlag Berlin Heidelberg.