Exploring the inter-rater reliability among experts and non-experts in video-based assessment of psychomotor nursing skills: a pilot study

Abstract

Nursing education focuses on mastering psychomotor skills. Students learn through feedback on their performance, based on observational assessments by nursing lecturers. This pilot study explores the inter-rater reliability of asynchronous video-based assessments by both experts and non-experts in nursing. Venepuncture performance by nursing students is recorded, and annotated by six assessors with a health sciences background, three with a nursing degree and three without. Inter-rater reliability is examined using a fully crossed design, analysing the intraclass correlation coefficient (ICC) for both consistency and absolute agreement of single and average measures. Based on 52 annotated videos, ICC single measures range between poor to good reliability among the six assessors. ICC average measures indicate excellent inter-rater reliability for assessors with nursing expertise and good reliability for those without. While these findings suggest that video-based assessment of psychomotor nursing skills for novice nursing students has potential, further validation is needed to confirm consistency across a broader range of assessors. Both lecturers with and without nursing expertise demonstrated the ability to assess venepuncture through video recordings, though proper training remains essential to improve scoring consistency, particularly for critical criteria. This study supports the use of video recordings for assessments, highlighting the benefits of flexibility, detailed review of student performance, and reduced potential for assessor bias. Involving lecturers from different health sciences backgrounds can help distribute the workload more efficiently in nursing education.