Methodological considerations for evaluating scale-up programmes in healthcare: A methods review

Abstract

Background The ultimate goal of many research projects is to achieve sustained improvements in health outcomes at population level. Scale-up refers to the integration of an effective intervention in routine practice and policy. Pertinent questions pertain to the appropriate study design in evaluating scale-up success. Methods A methodological review was conducted to determine how the scale-up of evidence-based interventions in healthcare can be evaluated. Specifically, we examined (i) appropriate research designs; (ii) outcomes and measures, endpoints; and (iii) key methodological considerations. Databases were searched and supplemented with hand searching journals and screening references and prospective citations. A narrative synthesis of included studies was produced. Results Several pre-scale-up design considerations were identified, including the need to assess the strength of the evidence base, develop a programme theory to guide measurements, and conduct a contextual analysis to inform implementation determinants. Evaluating baseline performance was recommended to define improvement targets, while scalability assessments were advocated to evaluate whether the intervention can be expanded with success. For scale-up evaluation, multiple evaluation domains and design considerations were identified. Adoption was described as the intention, decision, or action to try an intervention, which can be surveyed and supplemented with interviews to understand adoption decisions and processes. Coverage and reach were used to assess expansion of scale-up, defined respectively as the proportion of organizations implementing and the proportion of the target population reached, ideally tracked longitudinally with predefined targets. Routinely collected information, including insurance/billing data or administrative data, and survey designs can be used. Institutionalization referred to the integration of interventions into existing systems and structures. Recommended methods included stakeholder interviews, policy document reviews, and surveys with implementers, providers, and policymakers. Costs related to scale-up go beyond direct implementation costs and included broader system costs and costs associated with institutionalization. Evaluating effectiveness was considered important, including exploring variation across subgroups, providers, and regions. Design recommendations included predominantly nonexperimental designs, using pre-post and time series designs. Monitoring fidelity and adaptations were advocated, e.g. using observations. Across domains, mixed-methods approaches were recommended to capture both outcomes and the mechanisms through which they were achieved, recognizing scale-up as an iterative and adaptive process. Conclusion Longitudinal, adaptive, and mixed methods designs are needed to capture real-world implementation dynamics. We propose an initial conceptualization of scale-up success, defined by the main dimensions of coverage, reach, effectiveness, and institutionalization, which are contingent on cost of scale-up, adoption, fidelity, and adaptations.