Outcome imbalance in clinical trials: Mitigating loss of statistical power

Abstract

Background: Although cognitive changes are typical primary outcomes in alzheimer's disease clinical trials, a significant proportion of participants will remain stable throughout a trial. Furthermore, randomization does not guarantee equal rates of decline between placebo and treatment arms. This outcome imbalance between the arms can reduce a trial's power and/or lead to erroneous conclusions on treatment efficacy. Objectives: We used machine learning models to identify individuals as likely decliners or likely non‐decliners. We aimed to demonstrate that these prognostic labels can be used to increase a trial's power and mitigate the impact of outcome imbalance in two ways: post‐randomization subgroup analysis of likely decliners and pre‐randomization enrichment of likely decliners. Methods: We trained a machine learning model to classify decliners and non‐decliners on 1329 individuals with mild cognitive impairment or alzheimer's dementia from the ADNI (adni.loni.usc.edu) and NACC (naccdata.org) datasets. Decliners were defined as individuals who had increased CDR‐SB scores at 24 months of follow‐up compared to baseline, and the prevalence of decliners in this sample was 65%. Age, sex, APOE4 status, and gray matter volumes of brain regions extracted from MRI at baseline were used as input features. The model was trained and tested with nested 5‐fold cross‐validation. To assess the probability of outcome imbalance across placebo and treatment arms, we simulated 100,000 trials by randomly assigning individuals to placebo and treatment groups (n=250 per arm, based on a Phase 2 trial by Swanson et al, 2021, Alzheimers Res Ther), while also stratifying for APOE4 and diagnosis, and we measured the differences in prevalence of decliners and non‐decliners across the arms. We then assessed the impact of such imbalance on power to detect a 25% treatment effect across 1000 simulated trials at various levels of imbalance (ranging from 1% to 5% more decliners in the treatment arm). Furthermore, we studied whether covariate adjustment and enrichment with likely decliners predicted by the machine learning model can mitigate the loss in power associated with that imbalance. Results: The probability of observing at least a 5% outcome imbalance across arms (i.e. 5% more or fewer decliners in the treatment compared to placebo) was 22.4%. A perfectly balanced trial obtained a mean (± std) power of 89.2 ± 17.9% to detect a 25% treatment effect, while a trial that contained 5% more decliners in the treatment arm obtained only 73.8 ± 28.3% power, resulting in a drastic loss of 15% power. Covariate adjustment on common prognostic factors (e.g. APOE4 status, baseline diagnosis, baseline outcome score) increased power across levels of imbalance (97.2 ± 9.4% and 87.8 ± 21.5% in a perfectly balanced trial and a trial with 5% imbalance, respectively), but a 10% reduction in power still occurred at 5% imbalance. A subgroup analysis of the likely decliners (excluding the likely non‐decliners) identified by the machine learning model increased power regardless of imbalance (balanced: 92.6 ± 14.1%; 5% imbalance: 85.2 ± 21.2%), despite the reduction in sample size (63% of the original size of 250 per arm), compared to using data from the full sample. A subgroup analysis of likely decliners with covariate adjustment achieved even greater power (balanced: 97.4 ± 8.2%; 5% imbalance: 92.2 ± 16.0%). Enriching a trial at the original sample size with 250 likely decliners per arm increased the power (balanced: 97.0 ± 9.4%; 5% imbalance: 92.1 ± 16.9%). We observed that the 5% imbalance resulted in only a 5% power reduction when using the enriched population, compared to the 15% reduction on the unenriched population. Finally, covariate adjustment on an enriched sample with 250 likely decliners per arm obtained the greatest statistical power (balanced: 99.1 ± 4.7%; 5% imbalance: 96.2 ± 11.8%). Compared to using the unenriched population where a 5% imbalance resulted in a 10% power reduction, using the enriched population led to a 3% power reduction, rendering the statistical analysis robust to imbalance. Conclusion: An outcome imbalance across trial arms as little as 5% significantly reduces a trial's power to detect a treatment effect. The likelihood of enrolling imbalanced samples is approximately 1 in 5 (for a 5% discrepancy between the placebo and treatment arms) for a typical Phase 2 trial with 250 individuals per arm, despite randomization and balancing for common prognostic factors. Therefore, trials need additional strategies to enroll balanced arms. Our prognostic model can prevent imbalance by identifying individuals as likely decliners and likely non‐decliners prior to randomization. Enriching with likely decliners can mitigate the loss in power due to imbalance and boost power when there is perfect balance. Combining our enrichment strategy of selectively enrolling likely decliners with covariate adjustment enables even greater statistical power.