Caught up in the rhetoric of technology, data, and media companies? Problematic practices in dealing with nonscientific usage data

Abstract

Communication and media researchers, as well as scholars from related disciplines, are increasingly asking how social practices are measured, analyzed, and rcprcsent-cd through data or digital traces. However, there is little research on the various ways in which communication and media research itself uses such nonscientific usage data. This paper shows that usage data from technology, data, market research, and media companies are in many cases opaque and often insufficiently scrutinized and contextualized by scholars. There is too little critical discussion and analysis as to what different metrics and rankings reveal and what commercial interests and strategies arc driving them. The analysis at hand identifies and discusses five problematic ways of dealing with nonscientific usage data: A) lacking contextualization of what usage metrics and rankings indicate, B) lacking contcxtualization of rankings as strategic business tools, C) unsubstantiated inference from usage data to macro phenomena, D) uncritical adoption of superlatives and generalizations created by vested interests, and E) use of estimated numbers as measured data. The analysis of these problematics sheds a critical light on communication and media research, the partial lack of ensuring scientific quality, and recent developments in science. It also opens up new questions and perspectives with which communication and media research can make an important contribution. A critical look at usage data is essential, especially in societies in which usage figures and rankings are omnipresent and powerful corporations largely control access to data.