Two-stage interrupted time series analysis with machine learning: evaluating the health effects of the 2018 wildfire smoke event in San Francisco County as a case study

Abstract

Randomized controlled trials (RCTs) are considered a key identification strategy for establishing causal relationships between exposures and outcomes. When evaluating the health impacts of extreme weather events, however, RCTs are generally infeasible due to ethical issues, costs, and the lack of a suitable control group. Quasi-experimental designs capitalizing on the timing of natural experiments, such as Interrupted Time Series (ITS), offer a valuable alternative to estimate causal effects when control groups are not available. This paper explores the application of a 2-stage ITS framework that compares traditional autoregressive integrated moving average (ARIMA) models and 2 machine learning algorithms: Neural Network Autoregressive (NNETAR) and Prophet-Extreme Gradient Boosting (XGBoost). As a case study, we assess the impacts of the 2018 wildfire smoke event on respiratory hospitalizations in San Francisco County, CA. We split the data into pre- and postevent periods to train and evaluate the models, perform cross-validation for hyperparameter tuning, and predict hospitalizations under the counterfactual scenario. Data and R code are provided for reproducibility. In the case study, the Prophet-XGBoost shows the best model performance and was used to generate the counterfactual trends. We estimate that the 2018 smoke event resulted in a total of 92 (95% empirical CI, 24-125) excess respiratory hospitalizations (12.5% of the observed hospitalization count during the event period). Our proposed approach offers a powerful tool for assessing the effects of extreme weather events and can be broadly applied to other epidemiological contexts, such as public health policy evaluation.