Validity of using a work habits scale for the daily evaluation of nurse anesthetists' clinical performance while controlling for the leniencies of the rating anesthesiologists

Abstract

Study objective Anesthesiologists can provide psychometrically reliable daily evaluations of certified registered nurse anesthetist (CRNA) work habits for purposes of the mandatory ongoing professional practice evaluation (OPPE). Our goal was to evaluate the validity of assessing CRNA work habits. Design Observational study. Setting Large teaching hospital. Measurements N = 77 anesthesiologists evaluated work habits of N = 67 CRNAs. The non-technical attribute of work habits was measured on a 6-item scale (e.g., 1 = “Only assumed responsibility when forced to, and failed to follow through consistently” versus 5 = “Consistently identified tasks and completed them efficiently and thoroughly”). One year of scores were used to assess validity. Each daily evaluation could also be accompanied by a written comment. Content analysis of comments was performed using two years of data. Statistical analyses were performed using mixed effects logistic regression, treating each anesthesiologist as a fixed effect to compensate for the leniency of their ratings. Main results The N = 77 anesthesiologists' response rate was 97.3%, obtained at a mean 2.93 days after the request. The internal consistency of the scale was large: Cronbach's alpha 0.952. Controlling the false discovery rate at 5.0%, among the 67 CRNAs, 8 were significantly below average (each P ≤ 0.0048) and 6 were above average (each P ≤ 0.0018). During the 6 months after CRNAs knew that their work habits scores would be used for OPPE, there were significant increases in the scores compared with the preceding 6 months (odds ratio 1.93, P < 0.0001). Greater CRNA's qualitative annual evaluation scores made by the chief CRNA, without knowledge of the work habit scores or comments, were associated with greater odds of the CRNA's leniency-adjusted work habit scores equaling 5.00 (odds ratio 1.53, P = 0.0004). Comments of negative sentiment made by the anesthesiologists were associated with greater odds of the leniency-adjusted work habit scores being < 5.00 (odds ratio 54.5, P < 0.0001). Even though the anesthesiologists were already providing information about work habits using the work habits scale, approximately half the comments of negative sentiment included the theme of work habits (92/153, 60.1%). Conclusions Reporting OPPE metrics are mandatory for the maintenance of clinical privileges of anesthesia practitioners in the USA. Basing such peer review on work habits can be quantitative, psychometrically reliable, and valid.