Assessing Competence With a Task Trainer: Validity Evidence for Novel Tracheostomy Care Skills Assessment Tool

Abstract

Introduction The purpose of this study was to develop a caregiver and healthcare provider assessment tool to evaluate essential tracheostomy skills using a simulated task trainer. Methods Three tracheostomy skill checklists were developed: closed suctioning, open suctioning, and tracheostomy change. Checklist items were developed based on institutional guidelines and a literature review. Items were revised based on iterative expert review and pilot testing. A total of 64 intensive care staff and 24 caregivers were evaluated using the checklists, of which 29 staff members and 4 caregivers were rated simultaneously by 2 raters to estimate interrater reliability. The relationships between checklist performance and staff demographics (experience and discipline) were calculated. A survey examining the selection of automatic fail items and minimum passing score was sent to 660 multidisciplinary staff members. Results Intraclass correlations were 0.93 for closed suctioning, 0.93 for open suctioning, and 0.76 for tracheostomy change. Staff performance only correlated with experience for the tracheostomy change checklist and was inconsistently associated with discipline (respiratory therapy vs nursing). A large, multidisciplinary survey with 132 of 660 respondents confirmed the selection of automatic fail items and minimum passing score. A total of 92.9% of the survey respondents agreed with a minimum passing score of 80%. Conclusions We developed 3 essential tracheostomy skill checklists with multiple sources of validity evidence to support their use in a simulation-based assessment of tracheostomy skills.