A procedure for detecting student profile patterns in a performance assessment

Abstract

This study investigates student score profiles of the mathematics component of the 1997 Ontario grade 3 assessment. In addition to an overall score, students are given scores on three knowledge or skill dimensions, and five scores on content strands. The purpose of this investigation was threefold: (a) to assess the extent to which student profiles contain differentially diagnostic information, (b) to examine classroom-level patterns in the student profiles, and (c) to develop alternative methods of analyzing profile data to gain classroom-level diagnostic information. The results show that 70% of the students have the same score on all three knowledge/skill categories (flat profiles) and thus provide no differentially diagnostic information. The profiles for the remaining 30% of the students consisted almost exclusively of contoured profiles in which there was a difference of only one unit between one of the categories and the other two. Using algorithms developed in this article, these profiles were used to assess the relative strengths and weaknesses at the classroom level, as well as examining within-classroom diversity. This investigation found that nearly 60% of classrooms had a relative weakness in comprehension, and 25% of classrooms displayed substantial diversity. Examination of profiles of the five content strands indicated that nearly 69% of the students also had either flat or contoured profiles, whereas the remainder had more complex patterns. Methodologies for interpreting relative strengths and weaknesses on the mathematics strands at the student and the classmom level are discussed.