Exploring subjectivity in competency-based assessment judgements of assessors

Abstract

Aim: The aim of this research was to measure variations in assessors’ judgements of a student dietitian's performance and to explore the influence of group discussion on their judgements. Methods: The assessments of a student's performance, as observed from a video recording of an authentic nutrition consultation, were measured pre- and post-group discussion by 26 experienced assessors using a mixed-methods questionnaire. The instrument included a validated 7-point visual analogue scale (VAS) rating (1 = novice; 7 = competent), a qualitative global description of performance and an assessor's confidence rating (1 = not at all confident; 10 = extremely confident). Scales were analysed descriptively and qualitative responses coded for key themes. Results: No agreement was found in assessors’ rating in either the pretest (median = 4, range = 5) or post-test (median = 4, range = 4); however, the discussion led 78% of participants (20/26) to change their VAS ratings (9/26) and/or confidence levels (16/26). Three themes emerged from the thematic analysis of the participants’ global descriptions of performance: (i) discourse supports assessors to justify their judgements, identify assumptions and learn from the observations of others; (ii) discourse leads assessors to more holistic judgements; and (iii) multiple sources of evidence and student reflections are necessary for credible judgement. Conclusions: This research questions the notion that ‘actual’ performance can be objectively measured and, rather, considers assessments as ‘interpretations’. This research calls for an integrated interpretivist student-centred approach to competency-based assessment.