Eyeballs On Science: Impact Is Not Just Citations, But How Big Is Readership?

Abstract

—A full evaluation of the impact of scientific publications needs to count readers who don't generate citations. But this is very difficult to do. I set up a scholarly foraging experiment to try to estimate readership for my own body of work: a personal web site with 'reprint' pdf files available for downloading and a web statistics program engaged to count these downloads. (I made the site topically rather than author-oriented to increase potential audience size.) Despite this, human users are difficult to count. There are a lot of bots, spiders, and other web programs that increasingly mimic human behavior (e.g., with IP-and chrono-camouflage), and humans' own behaviors are changing (e.g., reading papers online many times). From four years of data, after culling both automated activity (a fascinating ecosystem) and humans reading repeatedly online, my papers receive ~4000-8000 downloads per year, about an order of magnitude higher than the number of citations they receive annually. This is a conservative minimum, because these publications can be obtained from other sources as well. On average, this body of work may be being read at an approximately 10:1 download-to-citation ratio. At a more granular scale, downloads do not correlate with citations; there are some papers being downloaded at about a 100:1 ratio, and it turns out these are exactly how they were meant to be used (e.g., an instruction manual). How intensively someone reads a paper is of course highly variable, but this exercise gives us an idea of how broad our publications' impacts actually are. Citations alone don't come close to measuring it. Introduction The evaluation of scientific impact has dwelt heavily on the number of times a work is cited. This is not unreasonable, and citation statistics have been readily available for most scientific journals through data aggregators such as the Science Citation Index (Thomson-Reuters), Scopus (Elsevier), and Google Scholar. But we know that this does not fully describe how often a work is read. How many of us read scientific papers that we don't cite? Clearly, measuring our consumption of science is not fully done using citation metrics. And consider also that fellow research scientists are not our only audience. In communicating our science, a broader audience of students, instructors, and the public is also important. How frequently our works are used beyond simple citations is an important question to answer in the development of more comprehensive evaluation of impact. And a more complete evaluation of impact is useful both to authors and to their evaluators. How different is consumption from citations? And how might we measure this? In the transition from physical to electronic distribution of scientific publications, it became possible to partially account for the number of times individual articles were accessed. This ability probably achieved its earliest transparency when journals went entirely online and openly reported the number of times an article was viewed. These metrics have been rapidly improving, giving us an increased ability to see how often articles are viewed, downloaded, cited, and discussed or used in various electronic media (often termed 'altmetrics' and posted prominently with an online article). As individual authors, however, these metrics are not yet widely available to us, leaving patterns and degrees of non-cited use an open question.