Weaknesses of the Traditional View of Standard Setting and a Suggested Alternative

Abstract

In this paper, we expand the traditional perspective on standard setting to include the necessary antecedents to a genuinely valid setting of standards, and use that conceptual framework to propose a new foundation for standard setting. These necessary antecedents include (a) the definition of an underlying variable on which the “standard” will be set in a way that is designed to be suitable for that standard setting, (b) the selection of a qualitatively definable point on that variable that corresponds to “enough” for the standard to be met, (c) the development of a suitable procedure (“test”) and expression of its results in a suitable way to readily afford use in a standard setting procedure, and (d) the application of a suitable method for deciding the observable cut score that reflects attainment of the standard. From this new perspective, we critique two examples of the traditional approach, the “Modified Angoff” and the “Matrix method.” We then describe an approach consistent with the more broadly-based foundation, centered on the Construct-Mapping line of thinking. We give an example of this in a unidimensional context. This approach is then generalized to address multidimensional constructs. We also illustrate a software application that has been developed to facilitate this process. We conclude by discussing some consequences of adopting the new approach, and survey needed next steps in research and development.