P2PIRB: Benchmarking framework for P2PIR

Abstract

Peer-to-Peer (P2P) is an attractive paradigm for building distributed network applications. A particularly intriguing class of distributed applications consists in Information Retrieval (IR) systems. The issue of Peer-to-Peer Information Retrieval (P2PIR) is being tackled by researchers attempting to provide valuable insights and to propose solutions to use it successfully. Nearly, all published studies have been evaluated by simulation means, using well-known document collections (usually acquired from TREC). This practice leads to two problems: First, there is little justification in favour of the document distributions used by relevant studies and second, since different studies use different experimental benchmarks, there is no common ground for comparing the solutions proposed. In this paper, we propose Peer-to-Peer Information Retrieval Benchmarking (P2PIRB), a benchmarking framework for P2PIR. P2PIRB allows to distribute documents and queries according to various ways. This work marks the start of an effort to provide more realistic evaluation environments for P2PIR systems as well as to create a common ground to compare the current and future architectures. © 2010 Springer-Verlag.