Comparing two methods for detecting adverse event signals in observational data: Empirical bayes gamma poisson shrinker vs. tree-based scan statistic

Abstract

Background: DuMouchel's Empirical Bayes Gamma Poisson Shrinker (EBGPS) and the tree-based scan statistic (TreeScan) have shown potential in adverse event (AE) signal detection in observational data. These methods offer different operational characteristics; EBGPS provides a shrinkage based point estimate and TreeScan allows a simultaneous evaluation across all levels of a clinical tree. Objectives: Compare EBGPS and TreeScan AE signal detection findings using observational data. Methods: We evaluated the performance of EBGPS compared to TreeScan using electronic health data at 9 HMO Research Network sites. We present preliminary results of an example: pioglitazone. Diagnosis codes were grouped into a hierarchical clinical classification tree. Observed and expected (O/E) counts were created for each group to form a dataset for analysis. Signals of excess risk during prevalent drug exposure, as compared to unexposed time, were evaluated based on O/E counts within all levels of the tree, controlling for age, sex and site. For EBGPS, 2 Bayes screening criterion were used approximating excess risk of >1.0 and >=2.0; p-values of <0.05 and <0.001 were used for TreeScan. Results: EBGPS found 52 signals across all criterion and levels of the tree; 13 at the most granular and 5 at the most general level. TreeScan identified 34 signals; 8 at the most granular and 4 at the most general level. All signals were either known AEs or could be explained by confounding. All EBGPS signals were identified by TreeScan, plus an additional 10 to 20 depending on p-value criterion. Restricting to the conservative signaling criterion, EBGPS identified 14 signals and TreeScan 25, which included all EBGPS signals. Conclusions: In this example, TreeScan was more sensitive than EBGPS and identified all the same signals as EBGPS. Use of population-based observational data may help identify potential AEs that warrant further epidemiological investigation. AE signal detection using either EBGPS or TreeScan can expand the medication safety surveillance arsenal. Understanding issues of false positive/negative findings is crucial for all signal detection methods.