To scrape or not to scrape, this is dilemma. The post-API scenario and implications on digital research

Abstract

Introduction: This article aims to investigate the potential impact of restricted social data access on digital research practices. The 2018 Cambridge Analytica scandal exposed the exploitation of Facebook user data for speculative purposes and led to the end of the so-called “Data Golden Age,” characterized by free access to social media user data. As a result, many social platforms have limited or entirely banned data access. This policy shift, referred to as the “APIcalypse,” has revolutionized digital research methods. Methods: To address the impact of this policy shift on digital research, a non-probabilistic sample of Italian researchers was surveyed and the responses were analyzed. The survey was designed to explore how constraints on digital data access have altered research practices, whether we are truly in a post-API era with a radical change in data scraping strategies, and what shared and sustainable solutions can be identified for the post-API scenario. Results: The findings highlight how limits on social data access have not yet created a “post-Api” scenario as expected, but it is turning research practices upside down, positively and negatively. On the positive side, because researchers are experimenting with innovative forms of scraping. Negatively, because there could be a “mass migration” to the few platforms that freely grant their APIs, with critical consequences for the quality of research. Discussion: The closure of many social media APIs has not opened up a post-API world, but has worsened the conditions of making research, which is increasingly oriented to “easy-data” environments such as Twitter. This should prompt digital researchers to make a self-reflexive effort to diversify research platforms and especially to act ethically with user data. It would also be important for the scientific world and large platforms to enter into understandings for open and conscious sharing of data in the name of scientific progress.