Intelligent spider for Internet searching

Abstract

As the World-Wide Web (WWW) based Internet services become more popular, information overload also becomes a pressing research problem. Difficulties with searching on Internet get worse as the amount of information that available on the Internet increases. A scalable approach to support Internet search is critical to the success of Internet services and other current or future National Information Infrastructure (NII) applications. A new approach to build intelligent personal spider (agent), which is based on automatic textual analysis of Internet documents, is proposed in this paper. Best first search and genetic algorithm have been tested to develop the intelligent spider. These personal spiders are able to dynamically and intelligently analyze the contents of the users selected homepages as the starting point to search for the most relevant homepages based on the links and indexing. An intelligent spider must have the capability to make adjustments according to progress of searching in order to be an intelligent agent. However, the current searching engines do not have the communication between the users and the robots. The spider presented in this paper use Java to develop the user interface such that the users can adjust the control parameters according to the progress and observe the intermediate results. The performances of the genetic algorithm based and best first search based spiders are also reported.