Simulation-based objective assessment discerns clinical proficiency in central line placement: A construct validation

Abstract

Background: Central venous catheterization (CVC) is associated with patient risks known to be inversely related to clinician experience. We developed and evaluated a performance assessment tool for use in a simulation-based central line workshop. We hypothesized that instrument scores would discriminate between less experienced and more experienced clinicians. Methods: Participants included trainees enrolled in an institutionally mandated CVC workshop and a convenience sample of faculty attending physicians. The workshop integrated several experiential learning techniques, including practice on cadavers and part-task trainers. A group of clinical and education experts developed a 15-point CVC Proficiency Scale using national and institutional guidelines. After the workshop, participants completed a certification exercise in which they independently performed a CVC in a part-task trainer. Two authors reviewed videotapes of the certification exercise to rate performance using the CVC Proficiency Scale. Participants were grouped by self-reported CVC experience. Results: One hundred and five participants (92 trainees and 13 attending physicians) participated. Interrater reliability on a subset of 40 videos was 0.71, and Cronbach a was 0.81. The CVC Proficiency Scale Composite score varied significantly by experience: mean of 85%, median of 87% (range 47%-100%) for low experience (0-1 CVCs in the last 2 years, n = 27); mean of 88%, median of 87% (range 60%-100%) for moderate experience (2-49 CVCs, n = 62); and mean of 94%, median of 93% (range 73%-100%) for high experience (> 49 CVCs, n = 16) (P =.02, comparing low and high experience). Conclusions: Evidence from multiple sources, including appropriate content, high interrater and internal consistency reliability, and confirmation of hypothesized relations to other variables, supports the validity of using scores from this 15-item scale for assessing trainee proficiency following a central line workshop. © 2010 American College of Chest Physicians.