Rates of false flagging due to statistical artifact in CMS evaluations of transplant programs: Results of a stochastic simulation

Abstract

The recent CMS conditions of participation are based on risk-adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying poor-performing centers is unknown. In this stochastic simulation study, 1-year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by CMS, evaluating nine overlapping 2.5-year periods of simulated data. In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false-positive flagging was lowest in low-volume centers (2.5%) and highest in high-volume centers (16.2%). In another simulation where 5% of centers were assigned twofold risk ("poor-performing centers"), only 32% of poor-performing centers were correctly flagged. In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater-than-median risk, but only 32% of flagged centers were among the 5% with highest risk. Even after disregarding known covariate limitations to the risk adjustment models, statistical noise alone leads to spurious flagging of many adequately-performing transplant centers, yet the methods used by CMS fail to flag most centers with true elevated risk. This simulation study of CMS evaluations of transplant programs shows that statistical noise plays a large role in determining which centers are flagged for poor outcomes. See editorial by Axelrod et al on page 1947. © Copyright 2013 The American Society of Transplantation and the American Society of Transplant Surgeons.