A rule-based approach for automatically extracting data from systematic reviews and their updates to model the risk of conclusion change

Abstract

Few data-driven approaches are available to estimate the risk of conclusion change in systematic review updates. We developed a rule-based approach to automatically extract information from reviews and updates to be used as features for modelling conclusion change risk. Rules were developed to extract relevant information from published Cochrane reviews and used to construct four features: the number of included trials and participants in the reviews, a measure based on the number of participants, and the time elapsed between the search dates. We compared the performance of random forest, decision tree, and logistic regression to predict the conclusion change risk. The performance was measured by accuracy, precision, recall, F1-score, and area under ROC (AU-ROC). One rule was developed to extract the conclusion change information (96% accuracy, 100 reviews), one for the search date (100% accuracy, 100 reviews), one for the number of included clinical trials (100% accuracy, 100 reviews), and 22 for the number of participants (97.3% accuracy, 200 reviews). For unseen reviews, the random forest classifier showed the highest accuracy (80.8%) and AU-ROC (0.80). All classifiers showed relatively similar performance with overlapping 95% confidence interval (CI). The coverage score was shown to be the most useful feature for predicting the conclusion change risk. Features mined from Cochrane reviews and updates can estimate conclusion change risk. If data from more published reviews and updates were made accessible, data-driven methods to predict the conclusion change risk may be a feasible way to support decisions about updating reviews.