Intelligent automated navigation through the deep web

Abstract

The DeepWeb, considered as the amount of built-on-demand (non pre-built)Web pages has become a very important part of the Web, not only because of its enormous size (it might considered that it is significantly bigger that the Superficial pre-built Web [12]), but because these Web pages usually contain customized information extracted from databases according to specific user’s requests. These pages are commonly robot-unreachable, usually requiring a login identification process or filling in some forms. Since pages within the Deep Web must be obtained within a navigation process, it is common that a single URL may not be enough for reaching them, so full navigation paths need to be stablished, usually by starting at a well-known URL and following some links and filling in some forms. On the other hand, Web Intelligence in a Web client might be considered as the property of properly combining several distributed data for solving a specific problem. Automated navigation through the Deep Web needs intelligence in order to reach relevant data which can be further computed. Web automated navigation involves both inter-document and intra-document navigations. Without intelligence at any of these two, Web clients can not stablish proper Web Navigation paths to those relevant data. This article presents an approach to formalize specifications of automated navigation on the Deep Web. These formalization has been expressed both graphically and textually in a combination of two languages for defining intelligent Web navigation behaviours at Web clients. Running examples of programs developed with these languages have been successfully developed and tested on legacy well known Web sites with low cost and a relatively high robustness.