An adaptive and scalable middleware for distributed indexing of data streams

Abstract

We are witnessing a dramatic increase in the use of data-centric distributed systems such as global grid infrastructures, sensor networks, network monitoring, and various publish-subscribe systems. The visions of massive demand-driven data dissemination, intensive processing, and intelligent fusion in order to build dynamic knowledge bases that seemed infeasible just a few years ago are about to come true. However, the realization of this potential demands adequate support from middleware that could be used to deploy and support such systems. We propose a peer-to-peer based distributed indexing architecture that supports scalable handling of intense dynamic information flows. The suggested indexing scheme is geared towards providing timely responses to queries of different types with varying precision requirements while minimizing the use of network and computational resources. Our solution bestows the capabilities provided by peer-to-peer architectures, such as scalability and load balancing of communication as well as adaptivity in presence of dynamic changes. The paper elaborates on database and peer-to-peer methodologies used in the integrated solution as well as non-trivial interaction between them, thereby providing a valuable feedback to the designers of these techniques. © Springer-Verlag Berlin Heidelberg 2004.