Copyright is held by the author/owner(s). FWCS2010, April 26, 2010, Raleigh, USA. Crowdsourcing Real-Time Research Trend Data Victor Henning Mendeley 144a Clerkenwell Road London EC1R 5DF +44 207 713 8486 victor.henning@mendeley.com Jason J. Hoyt Mendeley 144a Clerkenwell Road London EC1R 5DF +44 207 713 8486 jason.hoyt@mendeley.com Jan Reichelt Mendeley 144a Clerkenwell Road London EC1R 5DF +44 207 713 8486 jan.reichelt@mendeley.com ABSTRACT This paper proposes a demo of Mendeley at FWCS2010. Mendeley is a research workflow and collaboration tool, which crowdsources real-time research trend information and semantic annotations of research papers in a central data store. We describe how Mendeley���s data can overcome some of the weaknesses of traditional citation-based impact factor metrics and tag-based semantic databases. In the 12 months since its public launch, Mendeley has captured information on 13 million research papers, with its database doubling every 12 weeks. Categories and Subject Descriptors H.3.7 [Information Storage and Retrieval]: Digital Libraries ��� collection, dissemination, standards, user issues. General Terms Management, Measurement, Design, Standardization. Keywords Usage-Based Impact Measurement, Crowdsourcing, Article-level Metrics, Journal Impact Factor, Real-Time, Research Trends, Scientific Databases 1. THE SHORTCOMINGS OF CITATION- BASED IMPACT METRICS Citation-based reputation metrics such as the Journal Impact Factor (JIF), the h-index or the g-index play an ever-increasing role in modern science [1, 2, 3]. As seemingly objective measures of academic impact and performance, they are used to determine career progression, post-doc positions, tenure, and grant funding. Pressure on scholars to perform well according to these metrics has mounted. So has the criticism leveled against such metrics. It has been argued that these metrics can lead to academics engaging in citation bartering, gratuitous authorship, and a general increase in aggressive, exploitative, and self-promotional behavior [4]. Citation-based metrics are also thought to tempt journal editors into gaming the system using techniques that inflate their JIF. This includes only accepting papers expected to receive a higher number of citations, encouraging self-citations, and publishing review articles in place of research articles. From a methodological perspective, critics point out that citation counts are context-free, i.e. a citation is counted as positive even if a paper was cited in a negative context. Moreover, the Gini coefficient of the citation distribution is extremely high. A small fraction of all papers garner the majority of all citations, while the majority of all papers are never cited at all [5]. This also implies that a single highly cited article could inflate a JIF. Another major problem for JIFs is the arbitrary two-year window within which citations are measured, which favors fast-evolving disciplines. The h-index, similarly, is arbitrarily bound by the number of papers a researcher has published ��� so a young researcher with a few high-impact publications will still have a low h-index. Finally, there is evidence that only 20% of all papers cited have actually been read by the authors citing them [6]. 2. USAGE-BASED IMPACT METRICS ON MENDELEY Our demo at FWCS2010 will exhibit alternative, usage-based impact metrics which could potentially alleviate many of the problems associated with traditional citation-based metrics. Our usage-based metrics rely on a distributed measurement of article readership on Mendeley [7], a desktop- and web-based research management and collaboration tool. Mendeley Desktop, a free and cross-platform desktop application, automatically extracts metadata, full-text and cited references from research papers to minimize manual data input when setting up a local research paper database. It then enables researchers to manage, tag, full- text search, read and annotate PDF documents, share research papers with colleagues, and create bibliographies in word processors and text editors. Users can sync their libraries and annotations to the companion website, Mendeley Web, and manage them online. In this way, Mendeley Web has accumulated data on more than 13 million research papers and 150 million citations, by more than 100,000 users, in the first 12 months after its public launch. With Mendeley���s research paper database doubling in size roughly every 12 weeks, it is on track to surpass Thomson Reuters��� Web of Science catalogue of 40m full-text documents and 700m citations at some point this year. Our starting point for usage-based metrics is to track the pervasiveness of research papers in Mendeley user libraries, i.e. whether they are present on the computers of a wide-ranging, distributed sample of academics. Preliminary investigation suggests that, for example, the correlation between Thomson Reuters��� ISI citation count of the ���Top 5 Biology Papers of 2009��� and their corresponding readership number on Mendeley is r=.76 [8]. More encompassing correlation statistics will be presented during our demo at FWCS2010. Readership metrics can be seen as a measure of the popularity or awareness that a paper ��� and by association, its author, publication journal, and topic ��� is enjoying. A second, more fine- grained usage metric which Mendeley will begin to start tracking by FWCS2010 is the actual time the users spend reading each

research paper in Mendeley���s integrated PDF viewer, and the number of repeat readings per paper. This is a measure of the intensity with which the paper (its author, publication journal, topic, respectively) is being examined. A major advantage of such usage-based metrics is that they are available immediately on a ���per article��� basis. Usage-based metrics let authors track how readership of their individual papers is evolving in real-time, and the article���s impact can evolve independently of the journal and its impact factor. To better understand the readership, Mendeley also collects anonymous demographic information alongside the usage metrics. This information is presented to users in different segments such as geographic region, academic discipline, or junior versus senior faculty. Mendeley���s usage-based metrics let researchers retrieve the ���hottest��� papers for each topic (as marked by user-generated tags assigned to papers), the ���hottest��� tags in each academic discipline (to spot emerging research trends), or up-and-coming authors with the highest percentage growth in readership in the past month. By looking at longitudinal trend data, scholars might be able to assess whether a paper, topic or theory is steadily gaining followers, is subject to a sudden ���hype,��� or is already on the decline again. 3. REFERENCES [1] Garfield, E. The history and meaning of the journal impact factor. JAMA 295, 1 (2006), 90-93. [2] Hirsch, J. An index to quantify an individual's scientific research output. PNAS 102, 46 (2005), 16569-72. [3] Egghe, L. Theory and practise of the g-index. Scientometrics 69, 1 (2006), 131-51. [4] Lawrence, P. Lost in publication: how measurement harms science. ESEP 8, (2008), 9-11. [5] Weale, A., Bailey, M., and Lear, P. The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Medical Research Methodology 4, 1 (2004), 14. [6] Simkin, M. and Roychowdhury, V. Do you sincerely want to be cited? Or: read before you cite. Significance 3, 4 (2006), 179-181. [7] http://www.mendeley.com. [8] Henning, V. The Top 10 Journal Articles Published in 2009 by Readership on Mendeley. Mendeley Blog (2010), http://www.mendeley.com/blog/academic-features/the-top- 10-journal-articles-published-in-2009-by-readership-on- mendeley/.