Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms

  • Shrivastava* V
N/ACitations
Citations of this article
4Readers
Mendeley users who have this article in their library.
Get full text

Abstract

World Wide Web (WWW) is a massive resource of information that contains data in zeta bytes. Large number of users surfing Internet for various purposes need a quick response from web. Search engines are preferable, easily available and reliable source to retrieve data very rapidly with minimum resources. Search engine uses Web Crawler to search abundant data from web. It is a program that automatically passes through hypertext structure of the web and access webpage and hyperlink within it. Web crawling is a technique that focuses on obtaining relevant and fruitful links. To make the traversing accurate and productive, a properly designed web crawling algorithm is required. Web crawler and crawling techniques are kept secret as it is a hidden formula for the success of any search engine. In this paper, we have presented various web crawling algorithms along with their pros and cons to be utilized towards web search. Also we have proposed a novel web crawling algorithm to enhance the security and improving the performance of web crawler using single threaded and multi-threaded crawling algorithms.

Cite

CITATION STYLE

APA

Shrivastava*, V. (2019). Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms. International Journal of Recent Technology and Engineering (IJRTE), 8(4), 11642–11651. https://doi.org/10.35940/ijrte.d9430.118419

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free