Assessing evidence based on scale can be a useful predictor of policy outcomes

Abstract

With growing interest in more formalized applications of scientific evidence to policy, there are concerns about what evidence is selected and applied, and for what purpose. We present an initial argument that scale of evidence could be used in policy decisions in ways that can usefully predict effectiveness of policy interventions. This is valuable given that, as we show using a survey of of 251 policymakers, there is no single type of evidence (e.g., RCTs, systematic reviews, surveys) that is "best" to all policymakers or all policy domains. By simply rating the "level" of studies' size and scope used to inform policies, we show how high levels of evidence were more strongly associated with better (i.e., intended) outcomes across 82 policies. The rate of policies achieving intended outcomes ranged from 38%, when no evidence was available prior to the policy, to 78%, when large-scale evidence existed prior to implementation. Though these findings are encouraging, this piece is largely meant to argue for, not universally validate, a simple approach to assess evidence appropriately when making policy decisions. Instead, we argue that using this approach in combination with other ratings may better serve applications of evidence to achieve better outcomes for populations.