Using ORCID, DOI, and Other Open Identifiers in Research Evaluation

Abstract

An evaluator's task is to connect the dots between program goals and its outcomes. This can be accomplished through surveys, research, and interviews, and is frequently performed post hoc. Research evaluation is hampered by a lack of data that clearly connect a research program with its outcomes and, in particular, by ambiguity about who has participated in the program and what contributions they have made. Manually making these connections is very labor-intensive, and algorithmic matching introduces errors and assumptions that can distort results. In this paper, we discuss the use of identifiers in research evaluation-for individuals, their contributions, and the organizations that sponsor them and fund their work. Global identifier systems are uniquely positioned to capture global mobility and collaboration. By leveraging connections between local infrastructures and global information resources, evaluators can map data sources that were previously either unavailable or prohibitively labor-intensive. We describe how identifiers, such as ORCID iDs and DOIs, are being embedded in research workflows across science, technology, engineering, arts, and mathematics; how this is affecting data availability for evaluation purposes: and provide examples of evaluations that are leveraging identifiers. We also discuss the importance of provenance and preservation in establishing confidence in the reliability and trustworthiness of data and relationships, and in the long-term availability of metadata describing objects and their interrelationships. We conclude with a discussion on opportunities and risks for the use of identifiers in evaluation processes. In evaluation studies, we try to understand cause and effect. As research evaluators, our goal is to determine whether programs are effective, what makes them effective, what adjustments would make them more effective, and whether these factors can be applied in other settings. We start off with lofty goals and quickly descend into the muck and mire: the data-or lack thereof. In many cases, programs do not have clearly stated goals. Even when goals are stated, frequently data were not collected to monitor progress or outcomes. From the perspective of a research scientist, this approach is backwards. Researchers start with a hypothesis, develop a study process with specific data collection and controls, and then analyze the data to test whether their hypothesis is supported. Nevertheless we soldier on (one approach is described by Lawrence, 2017). Evaluators work with research program managers to develop frameworks to assess effectiveness. These frameworks, usually in the form of logic models, help establish program goals, and focus the questions to be addressed in the evaluation. Again, from lofty goals, we have to narrow and winnow our expectations based on the available data (Lane, 2016). Many program evaluations