Optimized focused Web Crawler with Natural Language Processing based relevance measure in bioinformatics web sources

24Citations
Citations of this article
26Readers
Mendeley users who have this article in their library.

Abstract

In the fast growing of digital technologies, crawlers and search engines face unpredictable challenges. Focused web-crawlers are essential for mining the boundless data available on the internet. Web-Crawlers face indeterminate latency problem due to differences in their response time. The proposed work attempts to optimize the designing and implementation of Focused Web-Crawlers using Master-Slave architecture for Bioinformatics web sources. Focused Crawlers ideally should crawl only relevant pages, but the relevance of the page can only be estimated after crawling the genomics pages. A solution for predicting the page relevance, which is based on Natural Language Processing, is proposed in the paper. The frequency of the keywords on the top ranked sentences of the page determines the relevance of the pages within genomics sources. The proposed solution uses a TextRank algorithm to rank the sentences, as well as ensuring the correct classification of Bioinformatics web page. Finally, the model is validated by being compared with a breadth first search web-crawler. The comparison shows significant reduction in run time for the same harvest rate.

References Powered by Scopus

Basic local alignment search tool

78874Citations
N/AReaders
Get full text

Focused crawling: A new approach to topic-specific Web resource discovery

765Citations
N/AReaders
Get full text

Extracting Structured Data from Web Pages

524Citations
N/AReaders
Get full text

Cited by Powered by Scopus

An Automated Word Embedding with Parameter Tuned Model for Web Crawling

77Citations
N/AReaders
Get full text

LEARNING-based Focused WEB Crawler

27Citations
N/AReaders
Get full text

Keyword weight optimization using gradient strategies in event focused web crawling

15Citations
N/AReaders
Get full text

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Cite

CITATION STYLE

APA

Mani Sekhar, S. R., Siddesh, G. M., Manvi, S. S., & Srinivasa, K. G. (2019). Optimized focused Web Crawler with Natural Language Processing based relevance measure in bioinformatics web sources. Cybernetics and Information Technologies, 19(2), 146–158. https://doi.org/10.2478/cait-2019-0021

Readers' Seniority

Tooltip

PhD / Post grad / Masters / Doc 6

60%

Lecturer / Post doc 3

30%

Researcher 1

10%

Readers' Discipline

Tooltip

Computer Science 8

80%

Agricultural and Biological Sciences 1

10%

Linguistics 1

10%

Save time finding and organizing research with Mendeley

Sign up for free