Uncovering impacts: a case study in using altmetrics tools Jason Priem1, Cristhian Parra2, Heather Piwowar3, Paul Groth4, and Andra Waagmeester5 1 University of North Carolina at Chapel Hill priem@email.unc.edu 2 University of Trento parra@disi.unitn.it 3 National Evolutionary Synthesis Center hpiwowar@nescent.org 4 VU University Amsterdam p.t.groth@vu.nl 5 Maastricht University andra.waagmeester@maastrichtuniversity.nl Abstract. Growing scholarly use of Web tools present an opportunity to track alternative impacts along heretofore invisible paths like read- ing, bookmarking, and discussing. We present two tools, CitedIn and total-impact, that gather and report these and other “altmetrics” After discussing the tools features, we use a set of 214 articles from a national research center as a demonstration case study. We find that both tools present a meaningful number and variety of altmetrics in a form that could be used for immediate evaluation, and call for more research into the properties and validity of altmetrics. Keywords: altmetrics, scholarly communication, impact, tools 1 Introduction The future of scholarly communication is one in which a large part of scholarly communication is conducted online [3]. A key part of the scholarly commu- nication lifecycle is trying to understand the impact of work. The process of understanding impact helps scientists, science administrators and others both find, evaluate, and access scholarly products. Traditionally, this impact assess- ment has been done primarily through the tracking of formal citations. This is possible because citations counts, for all their occasional ambiguity [2], do reflect use of scholarly products. However, this reflection is of a restricted spectrum scholarly products are often used by scholars, and others, in ways that do not perturb the citation record [5]. Furthermore, traditional citation does not reflect the rapid nature of communications afforded by the Web. Thus, we need new approaches for measuring impact in this changed world. Indeed, because of the Web scholarly communication, formerly “under- ground” uses like reading, bookmarking, sharing, discussing, and rating are be- ginning to leave online traces. The are becoming visible on Web pages [8, 13],

on blogs [6], in downloads [1, 4], on social media like Twitter [9], and in social reference managers like CiteULike, Mendeley, and Zotero [7]. These alternatives to traditional citation analysis have been labeled altmetrics [11]. Altmetrics offer potential for gathering information on more diverse types of impact, from more diverse scholarly products, including blog posts, slides, datasets, or even tweets. They also have the important benefit of speed altmetrics typically accumulate in days or weeks rather than the years citations require. This is particular useful in as the research process increases pace where users of scientific content need to understand the impact of it rapidly. To begin to make practical use of altmetrics for measuring impact requires both a greater understanding of the properties and validity of these new metrics, and practical tools for obtaining them [10]. Others have begun the former [12] here we will pursue the latter, presenting two new tools for gathering and presenting altmetrics. 2 Tools for Altmetrics: CitedIn and total-impact CitedIn (http://citedin.org) and total-impact (http://total-impact.org) are open- source tools that receive as input a list of identifiers for scholarly products, and output a set of altmetrics for each product. CitedIn accepts only articles with PubMed IDs (PMIDs) total-impact accepts articles identified by PMID or DOI, but also datasets and slides using a variety of identifiers including URL, han- dle, and accession numbers. Both tools allow users to input identifiers manually CitedIn also offers a REST API, and total-impact lets users automatically popu- late the products list using items stored in Mendeley or Slideshare libraries. Once users have uploaded products, CitedIn and total-impact both use calls to open Web APIs to gather data about them CitedIn also caches available databases. As of September 25, 2011, the data sources used by each are listed in Table 1. In addition to gathering altmetrics from these sources, both tools also include some additional features. CitedIn lets users input and output data over a REST API, and also reports a “CI-number” that summarizes all almetrics activity in a single value. Total-impact offers persistent URLs for impact report pages the impact metrics can be refreshed over time. Both tools let users download results as structured text files for further analysis. Output pages for the tools are shown in Figures 1 and 2. 3 Case study: altmetrics for a national research center We used a set of 214 articles from the National Evolutionary Synthesis Center (NESCent) as a realistic test for the two tools. NESCent was interested in track- ing the impact of work they funded in a faster and more comprehensive way than citation analysis allowed – a typical use case for altmetrics. We entered the articles into CitedIn on August 14 2011, and into total-impact September 23 2011, then collected and analyzed the results. All 214 articles had DOIs, and so were able to be processed by total-impact. Only 174 articles had the PMIDs required by CitedIn, so the CitedIn sample