A citation analysis of Australian information systems researchers: Towards a New ERA?

Abstract

Citation analysis is a potentially valuable means of assessing the contributions of researchers, in Information Systems (IS) as in other disciplines. In particular, a combination of raw counts and deeper analysis of citation data can deliver insights into the impact of a researcher's publications on other researchers. Despite this potential, the limited literature in the IS discipline has paid very little attention to the use of citation analysis for this purpose. Meanwhile, the federal department responsible for education funding has convinced successive federal governments to develop research quality measures that can be used as a basis for differential funding. The Howard Government's proposed Research Quality Framework (RQF) has been abandoned, but a number of aspects of it survive within the Rudd Government's Excellence in Research for Australia (ERA) initiative. The ERA also appears likely to involve a highly formalised process whereby 'research groupings' within individual universities will be evaluated, with (as yet unclear) impacts on the distribution of research funding. Funding agencies have an interest in score-keeping, whether or not their enthusiasm is shared by Australian researchers. It is therefore highly advisable that Australian disciplines, and especially less well-established and powerful disciplines like Information Systems, achieve a clear understanding of their performance as indicated by the available measurement techniques applied to the available data. This paper reports on citation analysis using data from both the longstanding Thomson/ISI collection and the more recently developed Google Scholar service. Few Australian IS researchers have achieved scores of any great significance in the Thomson/ISI collection, whereas the greater depth available in Google Scholar provides a more realistic picture. Quality assessment of the Thomson/ISI collection shows it to be seriously inappropriate for relatively new disciplines generally, and for IS in particular. Yet the Howard Government had selected the Thomson/ISI collection as the data-source for citation analysis for the now-abandoned RQF, and the Rudd Government appears likely to re-select it for the ERA. It is critical to the ongoing acessibility of research funding by Australian IS researchers that steps be taken to ensure that decision-makers are made very well aware of the inadequacies of the ISI data collection, and that alternative, more appropriate processes for impact assessment be developed.