Defining "Good" and Building "Better" Social Catalogues? Framework for Measuring and Improving the Quality of Research Literature Catalogues

Abstract

Automatically aggregating millions of user-added items into a social (crowd sourced) catalogue involves developing a set of catalogue quality control mechanisms such as duplicate detection and content normalization. To assess the effects of these control mechanisms it is necessary to quantify catalogue quality, which we do through defining a set of quality metrics. We are interested in measuring how different control mechanisms affect the catalogue quality. To discover this we devised a framework for quantifying quantifying aspects of catalogue quality affected by different types of catalogue control mechanism. Concretely, we define a set of metrics to quantify four quality aspects (a) coverage; (b) accuracy; (c) relevance; and (d) stability. The presented framework is implemented in Mendeley where it is used for measuring the quality of their social research catalogue, a catalogue which contains hundreds of millions of articles con- tributed by over two million users.