User testing with assessors to develop universal rubric rows for assessing engineering design

Abstract

This paper describes the process of testing and refining modular rubric rows developed for the assessment of engineering design activities. This is one component of a larger project to develop universal analytic rubrics for valid and reliable competency assessment across different academic disciplines and years of study. The project is being undertaken by researchers based in the Faculty of Applied Science and Engineering at the University of Toronto. From January 2014 to June 2015, we defined and validated indicators (criteria) for engineering design, communication, and teamwork learning outcomes, then created descriptors for each indicator to discriminate between four levels of performance: Fails, Below, Meets, and Exceeds graduate expectations. From this rubric bank, applicable rows can be selected and compiled to produce a rubric tailored to a particular assessment activity. Here we discuss these rubrics within the larger context of learning outcomes assessment tools for engineering design. We tested draft rubrics in focus group sessions with assessors (teaching assistants and course instructors who assess student work in engineering design). We followed the testing with structured discussions to elicit feedback on the quality and usability of these rubrics, and to investigate how the assessors interpreted the language used in the indicators and descriptors. We asked participants to identify indicators they believed were irrelevant, redundant, or missing from the rubric. We also asked them to identify and discuss indicators and descriptors that were confusing. Finally, we asked them what changes they would recommend and what training materials they would find useful when using rubrics of this design. We analysed the consistency of assessor ratings to identify problematic rows and used qualitative feedback from follow-up discussions to better understand the issues that assessors had with the rubrics. Three of the six engineering design rubric items we tested showed evidence of rater inconsistency, uncertainty, and indecision. While some rubric rows received similar criticism from most participants, we also identified differences in assessors' rubric design preferences and in how they apply rubrics to evaluate student work. It also emerged that assessors have different conceptions of engineering design and the design process, and are confused when presented with unfamiliar terminology. Based on our interpretations, we identified changes we should make to improve the rubric structure and content. Our aim is to inform other educators who may be developing tools for the assessment of engineering design by sharing our methodology and discussing the feedback received. Engineering educators could adopt or adapt our user testing methodology to improve the usability of similar assessment tools. Our discoveries about rubric structure improvements could be explored further to define best practices in the design of universal rubrics. Our next steps include applying what we have learned to refine the rubrics and develop accompanying training materials. The refined rubric rows will be evaluated for inter-rater reliability, trialed in focus groups with undergraduate students, and deployed in academic courses.