Assessing the quality of reports of randomised trials: Implications for the conduct of meta-analyses

Abstract

Objectives: To examine the issue of quality assessment of randomised controlled trials (RCTs) included in meta-analyses. To provide empirically based recommendations on how to conduct meta-analysis with respect to quality assessment. Five projects were carried out to achieve these objectives. 1. A database of meta-analyses was developed that provided the majority of data for the remaining projects. 2. Journal editors, methodologists and systematic reviewers associated with randomly selected articles in the database were surveyed about their views on the assessment and reporting of quality of the primary trials included in meta-analyses. 3. The frequency of quality assessment and the methods used were investigated using a sample of meta- analyses (n = 240) from the main database. 4. The effect that the quality of RCTs included in a meta-analysis has on estimates of intervention effectiveness was analysed using a sample of meta-analyses (n = 11 covering 127 RCTs) from the database. 5. Guidelines were developed on the basis of the evidence obtained in the other projects. Data sources: A comprehensive list of studies were provided by an electronic search of databases including MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews (CDSR). Study selection: Meta-analyses were selected. The inclusion criterion was that the study combined (pooled) the overall results of RCTs included in the meta- analysis. Data extraction: Data extraction forms were used to extract the necessary data from the articles. Data extraction was completed in duplicate to reduce the chances of error. Inter-rater reliability was calculated before data extraction began. Data synthesis: Quantitative analysis was difficult because of the nature of the research questions and was conducted only for the study examining the effect of RCT quality on estimates of intervention effectiveness. The data for the searching study and the survey, and the descriptive data for the quality assessment study, are discussed mostly in a qualitative manner. Results: The overlap of articles and journals between MEDLINE and EMBASE was 80% and 87%, respectively. The database of 491 articles that was used comprised 455 meta-analyses identified by the MEDLINE search and 36 meta-analyses in the CDSR. Response rates from the survey were 78%, 74% and 59% for reviewers (n = 121), methodologists (n = 55) and editors (n = 63), respectively. Over 90% of respondents stated that assessment and reporting of quality of RCTs included in meta-analyses was very or somewhat important. The use of RCT design features such as inclusion criteria, and using quality assessments to conduct sensitivity analyses were the most frequently endorsed methods of incorporating the quality assessments into meta-analyses. Most respondents believed that guidelines on the assessment and reporting of the quality of randomised trials would increase the rigour of reporting of published meta-analyses and make interpretation easier. Of a sample of 240 meta-analyses, trial quality was assessed in 48% and in half of these data on the reproducibility of the assessments were provided. Of the meta-analyses that assessed quality, only 25% incorporated trial quality into the analyses. Masked and unmasked quality assessments were carried out on 127 RCTs included in 11 meta-analyses in the database. The assessment were made using a validated scale (1-5, higher scores indicate superior reporting) and individual components known to affect estimates of intervention effectiveness. Masked quality assessment provided significantly higher scores (mean = 2.74; standard deviation (SD) = 1.10) than unmasked assessments (mean = 2.55; SD = 1.20). Low-quality trials were associated with an increase of 34% in estimate of benefit (ratio of odds ratios (ROR) = 0.66; 95% confidence interval (CI): 0.52, 0.83) compared with high-quality trials. Trials using inadequate allocation concealment, compared with those using adequate methods, were also associated with an increased estimate of benefit of 37% (ROR = 0.63; 95% CI: 0.45, 0.88). The average treatment benefit across all trials was 39% (OR = 0.61; 95% CI: 0.57, 0.65). Including only trials with low quality scores increased this effect to 52% (OR = 0.48; 95% CI: 0.43, 0.54), whereas including only trials with high quality scores reduced the effect to 29% (OR = 0.71; 95% CI: 0.65, 0.77). Using all the trial scores as quality weights reduced the effect to 35% (OR = 0.65; 95% CI: 0.59, 0.71) and resulted in the least statistical heterogeneity. Conclusions: Indexing inconsistencies within and across databases pose challenges in searching for systematic reviews or meta-analyses. Our results suggest that it is necessary to search multiple databases to identify all relevant information. Journal indexers, authors and editors should collaborate to develop and implement criteria to help users of systematic reviews and meta-analyses identify relevant publications. The systematic reviewers, methodologists and journal editors surveyed believed that assessment of trial quality was important. This contrasts with the infrequent reporting of trial quality in published meta-analyses. Future studies should address the issue of quality assessment. Consistent reporting of the design features of RCTs may help to enhance the rigour and clinical interpretability of meta-analyses. Among a sample of meta-analyses from the database, individual components and scales were the methods most commonly used to assess trial quality. However when quality assessments were made, in most cases they were not incorporated into the analysis. This is important because the incorporation of quality assessments can alter the estimate of the benefit of intervention, regardless of which method of assessment is used. The results from these studies also suggest that certain characteristics of the design and execution of RCTs impact on the probability of bias, and further research is needed on this. Investigations are also needed to clarify the value of masking studies before quality assessment and to determine the advantages of the various approaches to incorporate quality assessments into the analyses. Until such empirical evidence is presented, the guidelines outlined below are a useful tool with which meta-analysts, editors, peer reviewers and readers can deal with issues pertaining to quality assessment of randomised trials included in a meta- analysis. Guidelines: The quality of all randomised trials included in a meta-analysis should be assessed. Masked quality assessment should be considered, and meta-analysts should report masking methods used or their reasons for rejecting masking. Primarily evidence-based components (e.g. allocation concealment, double-blinding, type of randomised trial) should be used to assess quality. Topic-specific items should be part of the quality- assessment process. Scales used for assessment should have been appropriately developed and evaluated. A component approach has the advantage that it can be topic-specific. However, there is no compelling evidence to recommend a component approach over a scale approach or vice versa. Meta-analyses should incorporate an estimate of quality assessment into the quantitative analysis as a 'first-line' sensitivity analysis.