A New Method for Analyzing Content Validity Data Using Multidimensional Scaling

Abstract

Validity evidence based on test content is of essential importance in educational testing. One source for such evidence is an alignment study, which helps evaluate the congruence between tested objectives and those specified in the curriculum. However, the results of an alignment study do not always sufficiently capture the degree to which a test adequately represents the intended content domain. In this study, we present and evaluate a method for analyzing alignment data that uses multidimensional scaling to determine whether the dimensions underlying subject matter experts' (SME) ratings of test items conform to the dimensions delineated in the test specifications. The results suggest this procedure provides an enhanced view of content domain representation by portraying the content similarities among items along continuous, rather than discrete, dimensions. The additional types of information provided by this procedure, in relation to that provided by traditional analysis of SME data, are discussed. © The Author(s) 2013.