Small-Study Effects in Meta-Analysis

Abstract

This chapter describes small-study effects in meta-analysis and how the issues they raise may be addressed. ``Small-study effects'' is a generic term for the phenomenon that smaller studies sometimes show different, often larger, treatment effects than large ones. This notion was coined by Sterne et al. [55]. One possible, probably the most well-known, reason is publication bias. This is said to occur when the chance of a smaller study being published is increased if it shows a stronger effect [3, 41, 52]. This can happen for a number of reasons, for example authors may be more likely to submit studies with ``significant'' results for publication or journals may be more likely to publish smaller studies if they have ``significant'' results. If this occurs, it in turn biases the results of meta-analyses and systematic reviews. There are a number of other possible reasons for small-study effects. One is selective reporting of the most favourable outcomes, known as outcome selection bias or outcome reporting bias [8, 9, 18, 61]. Another possible cause of small-study effects is clinical heterogeneity between patients in large and small studies; e.g., patients in smaller studies may have been selected so that a favourable outcome of the experimental treatment may be expected. In the case of a binary outcome, also a mathematical artefact arises from the fact that for the odds ratio or the risk ratio, the variance of the treatment effect estimate is not independent of the estimate itself [47]. This problem will be discussed in Sect. 5.2.2. Lastly, it can never be ruled out that small-study effects result from mere coincidence [42]. Empirical studies have established evidence for these and other kinds of bias [19, 42, 53]. There is a vast range of tests for small-study effects [4, 20, 24, 38, 43, 48], most of them based on a funnel plot which will be introduced in Sect. 5.1.1.