1215Considerations for using multiple imputation in propensity score-weighted analysis

Abstract

Background Here we present the problems encountered when combining multiple imputation to handle missing data, propensity score-weighting to adjust for confounding and bootstrap to produce a percentile confidence interval. We apply our considerations to a research project estimating the association between long-distance migration and post-traumatic stress disorder with data from a sample of Syrian asylum seekers in Denmark and a sample of Syrian refugees in Lebanon. The exposure was "long-distance migration" defined as having migrated to Denmark instead of Lebanon and the outcome, PTSD, was assessed using the "Harvard Trauma Questionnaire" part IV, giving a score from 1 to 4 with 2.5 being the commonly used cut-off-score for PTSD. In a propensity score-weighted analysis you first estimate the propensity, here the chance of long-distance migration, given a relevant set of predictors, Pr (E = 1| Z) for each individual in the study population, ê i , by a regression model. The effect of long-distance migration on the prevalence of PTSD among those who migrated to Denmark was then estimated as the prevalence among those who have migrated minus a weighted average of the prevalence of PTSD among refugees in Lebanon, using weights equal to ê i /(1 – ê i ). This requires a number of decisions including: Which covariates to include in the propensity score model? How these should be introduced into the model? How to deal with extreme weights? And how to calculate the standard error of the parameter of interest? As we had missing data in the covariates and PTSD status, we set out to combine the propensity score-weighted analysis with multiple imputation which gives unbiased estimates assuming ignorable missingness mechanism and correctly specified multiple imputation model. Briefly, in multiple imputation a number, K , of data sets with imputed values for the missing data are produced and analyzed as planned, resulting in K estimates of β which are combined, typically by taking taking the average, into a final estimate for β. When implemented, the imputation is done for each variable with missing data (a) specifying a regression model for the conditional distribution of the variable given the other (relevant) variables (b) using the observed data to estimate the parameters in this model (c) impute the missing values of the variable by simulating from the Bayesian posterior predictive distribution. The latter two are generally handled by the software implementation, however, it is well known that it may introduce bias if the missingness mechanism is not ignorable or if the multiple imputation model is not compatible with the substantive model of interest. This raised additional questions such as: What are the required assumptions of the missing data process to be ignorable? How should we do the multiple imputations? How to combine the multiple imputations with the propensity score analysis? How to find a valid confidence interval for the parameter of interest taking into account the modeling (and thus introduction of uncertainty) in both the propensity score estimation and multiple imputation? The theoretical background for these and additional considerations will be presented at the Early Career Workshop. Methods Here we outline our estimation algorithm. Based on the a priori analysis plan three propensity score models of increasing complexity were defined and three levels of weights truncation (no truncation, truncating at the 1st and 99th percentile, or truncating at the 5th and 95th percentile) were examined for covariate balance. Based on one multiple imputed data set for each of the three complexities of the propensity score model, the least complex model with the least amount of truncation to obtain acceptable balance, defined as absolute standardized difference of ≤ 0.10 on all covariates was chosen for the analysis. Since analysing the data from a frequentist perspective the "everywhere missing-at-random" and congenial model assumptions were examined and if not violated, the missing data was multiply imputed using the Substantive Model Compatible Fully Conditional Specification (SMC-FCS) implementation. The substantive model was the chosen propensity score model and for each partially observed covariate a "prediction model" was specified. The sampling interval between the imputations was decided based on plots of the parameter estimates against the sampling interval. For each of the multiply imputed data sets the propensity scores were computed, converted into weights and the weighted point estimate produced. The point estimate of interest was the average of all point estimates and the 95 percentile confidence interval was produced by subsequently bootstrapping the above steps a large number of times. The procedure is illustrated in Figure 1. 1215 Figure 1 Open in new tab Download slide Flow-chart of proposed methodology to combine multiple imputation and propensity score weighting. *The bootstrap is repeated multiple times, for example 1000, to be able to estimate the 95-percentile confidence interval. 1215 Figure 1 Open in new tab Download slide Flow-chart of proposed methodology to combine multiple imputation and propensity score weighting. *The bootstrap is repeated multiple times, for example 1000, to be able to estimate the 95-percentile confidence interval. Results A simple propensity score model with no interaction terms and weight truncation at 1st and 99th percentile obtained acceptable balance on all covariates and was chosen as our model. Unfortunately, our first choice of substantive model had to be modified due to computational/numerical problems by collapsing two levels of one of the substantive model covariates and two levels of one of the auxiliary covariate. The number of imputations was set to 10 and the convergence plots showed that a sampling interval between imputations of 40 was abundant. To compute the 95-percentile confidence interval 999 bootstrap estimates were produced. The procedure was split on three computer systems and the time to run 250 bootstrap estimates was from two to 10 hours. The analysis showed an increased prevalence amounting to 8.76 percentage point (95-percentile confidence interval: [-1.39; 18.62] percentage point) with little variation in the sensitivity analysis. Conclusions In this Early Career Workshop proposal we describe the statistical methodological considerations for combining propensity score-weighting for confounding control and multiple imputation of missing data. and discuss the assumptions underlying both of the statistical methods. In our approach, the substantive model of interest and covariates to include in the propensity score model was explicit. It has been suggested that "black box" or machine learning algorithms may provide reasonable propensity score-weights, however, at the cost of control over the substantive model which is paramount in fulfilling one of the assumptions of multiple imputation: a correctly specified substantive model of interest. We have based our propensity score model on the available evidence and subject matter knowledge, however, recognise the possibility of remaining bias, for example from residual confounding and from the collapsing of two levels of one of the substantive model variables. A procedure similar to the one we apply to combine propensity score-weighting and multiple imputation has been shown to theoretically give unbiased point estimates assuming ignorable missingness mechanism. When the missingness mechanism is not ignorable including a "missing-value indicator variable" in the data set may reduce bias, however, conversely, increase bias when the missingness mechanism is ignorable. We subjected every variable to careful examination and are satisfied that the "everywhere-MAR" assumption is not violated, however, we acknowledge that this is subject to discussion and cannot be guaranteed. We used bootstrap to produce a 95-percentile confidence interval taking into account uncertainty introduced by modelling in both the propensity score and multiple imputation step. "Rubin's rule" is the traditional choice when doing multiple imputation but does not account for the uncertainty introduced in the propensity score estimation and, thus, is not valid in theory. There is no clear evidence on what step to bootstrap when combining propensity score-weighting and multiple imputation. In our approach, we bootstrapped the entire procedure. to produce a confidence interval that accounts for all uncertainty introduced by modelling. Others suggest that bootstrapping after multiply imputing the data sets may produce similar results at lower computational expense, however, this may increase bias. Our methodology takes several hours to run on "standard" laptop computers and we experienced numerical problems with strata with relatively few observations. Going forward, we are eager to examine the sensitivity of our result to different methodologies for example using other g- methods such as g- computation or other multiple imputation methods such as machine learning algorithms. The produced point estimate and confidence interval could also be compared to alternative methods that lowers the computing time such as "Rubin's rules" or the recently proposed "von Hippel" method for using bootstrap in multiple imputation (though does not include propensity score modelling). Key messages In this Early Career Workshop proposal we make clear the considerations we had to go through to combine propensity score-weighting and multiple imputation for analysing observational data. The many choices are not trivial and gives rise to discussions of alternatives. [ABSTRACT FROM AUTHOR]