Examining Cultural Responsiveness in Large-Scale Assessment: The Matrix of Evidence for Validity Argumentation

Abstract

As cultural products, assessment systems, and instruments are a reflection of the cultures in which they originate and reproduce the characteristics of those cultures. Accordingly, cultural responsiveness in large-scale assessment concerns the entire process of assessment, not simply its tests or the analysis of test scores. This paper focuses on validity as argumentation in the testing of culturally diverse populations in both national and international large-scale assessment programs. It examines the intersection of m procedural assumptions (conceptual and methodological criteria that need to be met in order to validly, fairly test culturally diverse populations), and n components of the process of assessment. The paper explains the development and use of the matrix of evidence for validity argumentation. In this matrix, a cell [i, j] contains confirming and disconfirming evidence that Procedural Assumption i is met by the practices enacted and the artifacts used or generated in Assessment Process Component j. An argument of validity is constructed by integrating the information contained in the cells of the matrix. A series of examples illustrate how each procedural assumption intersects with each assessment process component in the matrix. These examples show the ubiquity of cultural issues in assessment. Attaining cultural responsiveness in large-scale national and international assessment programs entails a serious systemic, societal, global effort.