Electronic copy available at: http://ssrn.com/abstract=1088453 ACCESS, USAGE AND CITATION METRICS: WHAT FUNCTION FOR DIGITAL LIBRARIES AND REPOSITORIES IN RESEARCH EVALUATION? Chris Armbruster Research Associate, Max Planck Digital Library, Max Planck Society Invalidenstrasse 35, D 10115 Berlin www.mpdl.mpg.de Executive Director, Research Network 1989 www.cee-socialscience.net/1989 Abstract The growth and increasing complexity of global science poses a grand challenge to scientists: how to best organise the worldwide evaluation of research programmes and peers? For the 21st century we need not just information on science, but also meta-level scientific information that is delivered to the digital workbench of every researcher. Access, usage and citation metrics will be a major information service that researchers will need on an everyday basis to handle the complexity of science. Scientometrics has been built on centralised commercial databases of high functionality but restricted scope, mainly providing information that may be used for research assessment. Enter digital libraries and repositories: can they collect reliable metadata at the source, ensure universal metric coverage and defray costs? This systematic appraisal of the future role of digital libraries and repositories for metric research evaluation proceeds by investigating the practical inadequacies of current metric evaluation before defining the scope for libraries and repositories as new players. Subsequently, the notion of metrics as research information services is developed. Finally, the future relationship between a) libraries and repositories and b) metrics databases, commercial or non-commercial, is addressed. Services reviewed include Leiden Ranking, Webometrics Ranking of World Universities, Ranking Web of Repositories, COUNTER, MESUR, Harzing Publish or Perish, Hirsch Index, CiteSeer, Citebase, SPIRES, SSRN CiteReader, RePEc LogEc and CitEc, Scopus, Web of Science and Google Scholar. Keywords Scientometrics, webometrics, research evaluation, research assessment, citation metrics, usage metrics, access metrics, digital library, digital repository, open access

Electronic copy available at: http://ssrn.com/abstract=1088453 2 Table of contents 1. Introduction: Advancing science by metric research evaluation ��� Purpose and argument 2. Libraries, repositories and scientometrics: enhancing the differentiation of research programmes and the global integration of scientists? ��� Measuring impact: the bibliometrics of universities and departments ��� Measuring access: webometrics of universities and repositories ��� Opportunity and challenge for libraries and repositories: standardised descriptors for the online tracking of item, author and institution 3. Open science and research evaluation: repositories as base? ��� Institutional repositories for research evaluation? ��� Metric services of disciplinary repositories ��� Opportunity and challenge for digital libraries and repositories: setting standards for statistics 4. Metric evaluation: towards research information services ��� Citation metrics and the example of the h-index as research information service ��� Usage metrics and the example of library statistics ��� Metric batteries as research information service ��� Opportunity and challenge for digital libraries and repositories: building research information services 5. Conclusion: What relationship between digital libraries, repositories and metrics databases? ��� High functionality, limited value and restricted compatibility: commercial databases ��� Stifled potential: non-commercial databases ��� Strategic alternatives for libraries and repositories: club good or search engine? I would like to thank Fredrik Astrom, Lars Bj��rnshauge, Hakan Carlsson, Werner Marx, Emanuela Reale, Laurent Romary and Hermann Schier for extensive comment and criticism. For crucial input I also thank participants of a roundtable held on the subject of libraries, repositories and metric research evaluation at the University of Padua on 19 September 2007: Isidro Aguillo, Christoph Bruch, Johannes Fournier, Fred Friend, Wolfram Horstmann, John MacColl, Susanna Mornati, Sijbolt Noorda and Birgit Schmidt. A second roundtable was held at the University of Lund on 24 April 2008 with Frederik Astrom, Ahmed Hindawi, Wolfram Horstmann, John MacColl, Alain Monteil, Laurent Romary and John Wilbanks. Any remaining omissions and errors are mine.

3 1. Introduction: Advancing science by metric research evaluation A century of unprecedented growth and internationalisation of research, higher education and scientific publishing has led to a degree of complexity that poses a grand challenge to scientists: ��� How to keep track of very large amounts of information and tasks? ��� What tools to use in organising knowledge and people in pursuit of new scientific breakthroughs? ��� Which measures to take in enabling the further differentiation of research programmes while enhancing the global integration of researchers? The Internet has principally eased access to knowledge while enhancing collaboration at a distance. Yet, the Internet also adds to the complexity, and the rise of e-science will reinforce this trend. Tools for self-observation are important and urgently needed. Citation metrics have come to play an important role in many fields of the natural sciences, e.g. for hiring and tenure review as much as for decisions on to which journal to submit an article. Usage metrics are gaining ground in the social sciences. They are also relevant to research fields that have large numbers of users who are not also authors that will cite what they read. Moreover, the question whether earlier usage predicts later citations is being addressed. Citations and usage metrics are indicators for impact. Access metrics, by contrast, measure the online accessibility of research. This is not only a matter of volume, but also of quality. Access, usage and citation are distinct concepts, refer to different actions and are measured separately. However, commercialisation has led to an impasse over access to knowledge (e.g., the anti-commons of patents, the serials crisis) and politically instigated research assessments and excellence awards lead to the rigid stratification of institutions. While research assessment typically has taken the form of peer review ���by hand,��� government agencies have begun pushing for metric review as more efficient, impartial and objective. While political attention brings resources, it also carries the danger of arresting the further development of tools. Research assessment of institutions and individuals is one form of research evaluation, albeit a narrow one aimed at ranking and quality assurance. Science has principally always been a world system, but evidently international databases could only be built through private funding, e.g., by the Institute for Scientific Information (ISI, now part of the Thomson Reuters Cooperation). However, the commercial definition of the corpus and the metrics that can be derived from it has historically been very narrow. Internal coverage of the corpus is sufficient only in those fields in which English language journal articles have for a long time been the overwhelming means of communicating ideas and research results. Practically speaking, metrics and related tools for tracking and indexing are presently most developed for physics, chemistry and biomedicine. Computer scientists and economists have been building not-for-profit online services for their communities, but these are not yet comprehensive and reliable enough. Scientometricians have been reliant on commercial databases and tend to focus on performance measures. It may be surmised that this is due to a conjunction of

4 zeitgeist (with targets and performance measures as a means of policy implementation and league tables for everything) and a receptive audience among research assessors (including government departments). However, the focus on past performance, league tables and top-up funding for alleged world-class research brings with it the risk of transforming the differentiation of research programmes into a rigid institutional stratification. The resulting social inequality that will turn ranking into a rat race might well undermine the essential cultural norms of science such as communalism, universalism, disinterestedness and organised scepticism (Merton 1942). Current research evaluation is defined by political exigencies and constrained by commercial interests. However, so my argument, we need to understand that evaluation is something that researchers increasingly undertake on a daily basis. Moreover, due to the increasing global complexity of science we need new meta- level tools for self-observation. In this scenario, libraries and repositories will come to play an important role in developing tools and delivering data to the digital workbench of researchers. Purpose and argument My purpose is not to spuriously claim that digital libraries and repositories will deliver all the answers. The aim is principally to investigate the potential of new tools for research evaluation, focussing in the first instance on access, usage and citation metrics. Digital libraries have become involved in much more than the acquisition of scientific information, presenting and preserving materials produced by researchers, typically sharing them in open access (cf. California Digital Library for a university system or Max Planck Digital Library for a national research academy). At its most sophisticated, digital libraries bring materials to the researcher���s workbench and capture and collect the products (e.g., data, articles). Digital libraries are increasingly significant at institutional, national, regional and global level (cf. the Digital Library Federation in the US, European Digital Library or World Digital Library).1 Digital libraries include repositories and digital repositories are a new type of library. Repositories come in various forms, having an institutional, national or disciplinary focus. Several large disciplinary repositories have been built for physics (arXiv), biomedicine (Pubmed), computer science (Citeseer), economics, law and the social sciences more generally (RePEc, SSRN). Interestingly, in biomedicine the disciplinary and national form is combined with the setting up of UK Pubmed. France has most consistently opted for a national database (Hyper Article en Ligne ��� HAL), while the US, Germany, UK, Japan and Australia together account for more than half the worlds��� institutional repositories.2 Metric data may be aggregated to produce information about any unit associated with research ��� e.g., author, article, book, data set, funding, supervision, publication outlet, research team, institution and country. Provided that clean data 1 http://www.cdlib.org/ - http://www.mpdl.mpg.de/ - http://www.diglib.org/ - http://www.edlproject.eu/ - http://www.worlddigitallibrary.org/ 2 http://arxiv.org/ - http://www.pubmedcentral.nih.gov/ - http://citeseer.ist.psu.edu/ - http://repec.org/ - http://www.ssrn.com/ - http://ukpmc.ac.uk/ - http://hal.archives-ouvertes.fr/ - http://www.opendoar.org/

5 are obtained according to a global standard, a federation of repositories would offer a database that would be superior in every respect: size, variety and quality. If we were to consider only citation metrics, a truly global database would emerge as digital libraries and repositories worldwide adhere to standardised descriptors for author, institution and item. Not just journal articles may be tracked, but also a variety of other publications, such as books, book chapters, conference proceedings, thesis, working papers, and possibly even patents. The version of any publication could also be controlled, distinguishing for example working papers from post-prints. This would produce a much richer impact on research as well as a much more nuanced picture of science. A long and growing list of sceptical responses to metric research evaluation by individual researchers, journal editors and institutional leaders exists. As will become clear from the following analysis, current services do not respond yet to the grand challenge of enhancing the capacity of science for self-observation to such a degree that researchers can effectively reduce complexity in pursuit of the further advancement of knowledge. A popular line of argument is not to dismiss metric research evaluation outright, but to insist that experts (read: the senior scholars) must be the ultimate arbiters. However, on the grounds of logic and evidence there is little to suggest that individual experts or even a group of experts deliberating will reach a more informed understanding as to the importance or impact of research than the dispersed community that sets links, downloads and cites. This is not about the wisdom of the crowds, but based on the observation that an aggregate measure of access, usage and citation is a much more reliable predictor of influence than the opinion of experts. That said, the following argument does not engage the critics of metric research evaluation. It is assumed that we need metrics for the further advancement of science. In that vein, we must search for the best possible solution. Currently, it is far from self-evident that the optimal solution will be implemented. Furthermore, we cannot even be confident that solutions adopted will further the advancement of science rather than obstruct it. Neither the existing commercial nor the non- commercial models hold enough promise and so-called open access scientometrics is also full of problems. With those reservations in mind, the following systematic exposition of digital libraries and repositories in relation to scientometrics addresses the grand challenge of furthering the future growth and advancement of science. 2. Libraries, repositories and scientometrics: enhancing the differentiation of research programmes and the global integration of scientists? For metrics to be of utility for reading, writing and peer review, the sifting of applications, tenure review or the assessment of research groups, measures beyond stargazing are required. Because metrics lend themselves to rankings, there has been a marked tendency to construct top lists and focus all attention on the winners. While this is broadly compatible with the bestowal of recognition and reputation by peers, these kinds of rankings are one dimensional, about the past and often of research programmes long consolidated.