Combined use of web scraping and AI-based models for business applications: research evolution and future trends

Abstract

Online data are instrumental in knowledge generation and decision-making processes, particularly for companies aiming to glean insights for marketing strategies and service optimizations. Web scraping emerges as a pivotal technique for automatic data collection from the internet, crucial in today's competitive business landscape where online presence is paramount. Coupled with advanced artificial intelligence (AI) models, web scraping becomes even more powerful, enabling deeper analysis of unstructured data for enhanced decision-making. However, managing and analysing the diverse online information presents challenges. Through a bibliometric analysis conducted on 567 collected papers, this study aims to elucidate the evolving business landscape of integrating web scraping with AI-based models. In particular, it addresses key research questions regarding the evolution, business implications, and future trends of this integration. Results show a significant growth in research in this domain, with implications spanning sentiment analysis, natural language processing integration, data intelligence, and deep learning integration. The critical interpretation of the results has allowed for the definition of a future research agenda that proposes concrete directions for the development of more transparent, safe, sustainable and democratic technologies. The paper provides both theoretical and practical contributions. From an academic perspective, it offers insights into the effective utilization of AI-based models and web scraping integration for optimized scraping strategies in business, providing a clear overview of emerging research direction in this field. From a managerial perspective, this paper guides managers in adopting advanced AI-based models to positively impact their company's decision-making involving a strategic and interdisciplinary reflection on the responsible use of scraping and AI.