Abstract
Distributed Web crawlers have recently received more and more attention from researchers. Full decentralized crawler without a centralized managing server seems to be an interesting architectural paradigm for realizing large scale information collecting systems for its scalability, failure resilience and increased autonomy of nodes. This paper provides a novel full distributed Web crawler system which is based on structured network, and a distributed crawling model is developed and applied in it which improves the performance of the system. Some important issues such as assignment of tasks, solution of scalability have been discussed. Finally, an experimental study is used to verify the advantages of system, and the results are comparatively satisfying. © 2008 Springer-Verlag Berlin Heidelberg.
Author supplied keywords
Cite
CITATION STYLE
Zhu, K., Xu, Z., Wang, X., & Zhao, Y. (2008). A full distributed Web crawler based on structured network. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 4993 LNCS, pp. 478–483). https://doi.org/10.1007/978-3-540-68636-1_51
Register to see more suggestions
Mendeley helps you to discover research relevant for your work.
