Focused crawling using context graphs

  • Diligenti M
  • Coetzee F
  • Lawrence S
 et al. 
  • 142

    Readers

    Mendeley users who have this article in their library.
  • 312

    Citations

    Citations of this article.

Abstract

Maintaining currency of search engine indices by exhaustive crawling is rapidly becoming impossible due to the increasing size and dynamic content of the web. Focused crawlers aim to search only the subset of the web related to a specific category, and offer a potential solution to the currency problem. The major problem in focused crawling is performing appropriate credit assignment to different documents along a crawl path, such that short-term gains are not pursued at the expense of less-obvious crawl paths that ultimately yield larger sets of valuable pages. To address this problem we present a focused crawling algorithm that builds a model for the context within which topically relevant pages occur on the web. This context model can capture typical link hierarchies within which valuable pages occur, as well as model content on documents that frequently cooccur with relevant pages. Our algorithm further leverages the existing capability of large search engines to provide partial reverse crawling capabilities. Our algorithm shows significant performance improvements in crawling efficiency over standard focused crawling.

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

Authors

  • Michelangelo Diligenti

  • Frans Coetzee

  • Steve Lawrence

  • C. Lee Giles

  • Marco Gori

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free