Assessing the methodological quality of systematic reviews of interventions aimed at improving vaccination coverage using AMSTAR and ROBIS checklists

Abstract

Introduction: Systematic reviews (SRs) are the backbone of evidence-based health care, but no gold standard exists to assess their methodological quality. Although the AMSTAR tool is accepted for analyzing the quality of SRs, the ROBIS instrument was recently developed. This study compared the capacity of both instruments to capture the quality of SRs of interventions for improving vaccination coverage. Methods: We conducted a comprehensive literature search in the Cochrane Library and PubMed. Two reviewers independently screened the search output, assessed study eligibility, and extracted data from eligible SRs; resolving differences through consensus. We conducted Principal Component Analysis (PCA) in Stata 14 to determine similarities and differences between AMSTAR and ROBIS. Results: A total of 2322 records were identified through the search and 75 full-text publications were assessed for eligibility, of which 57 met inclusion criteria. Using AMSTAR, we found 32%, 60% and 9% of SRs to have high, moderate and low quality, respectively. With ROBIS, we judged 74%, 14% and 12% of SRs to have low, unclear and high risk of bias. PCA showed that SRs with low risk of bias in ROBIS clustered together with SRs having high-quality in AMSTAR, and SRs with high risk of bias in ROBIS clustered with low-quality SRs in AMSTAR. Conclusions: Our findings suggest that there is an association between methodological quality and risk of bias in SRs of interventions focused on improving vaccination coverage. Therefore, either AMSTAR or ROBIS checklists can be used to evaluate methodological quality of SRs in vaccinology.