Scalable Web Search by Adaptive Online Agents: An InfoSpiders Case Study

Abstract

The trend of the recentyears in distributed information environments is a good example of the life-like complexity that we expect to observeinmost aspects of information and computational science. The explosion of the Web and electronic mail, multiplying the number of information providers and consumers manytimes over and bringing the Internet inside the average home, has created formidable new opportunities and challenges in almost every area of computer and information science. In an e?ort to address such problems, researchers in arti?cial intelligence and information retrieval have already been successful in developing agent- based techniques to automate many tedious tasks and facilitate the manage- ment of the growing amounts of information ?ooding users. But the work has just begun. There is still much need for tools to assist users in ways that scale with the growth of the Web, and adapt to both the personal preferences of the user and the changes in user and environmental conditions. This chapter discusses an agent-based approach to building scalable in- formation searching algorithms. For systems designed to let users locate rel- evant information in highly distributed and decentralized databases, suchas the Web, we argue that scalability is one of the main limitations of the cur- rent state of the art. Given suchanambitious goal, it probably comes as no surprise that the solution proposed here draws on many ideas and issues discussed in other parts of this book: cooperation in multi-agent systems, information chain economy in rational agents, and spawning and securityin mobile agents.