Multi-strategy in the evaluation of health promotion community interventions: An indicator of quality

Abstract

There is a general agreement in the specialized literature on the need to design and conduct multi-strategy evaluation in health promotion and in social sciences. "Many community-based health interventions include a complex mixture of many disciplines, varying degrees of measurement difficulty and dynamically changing settings . . . understanding multivariate fields of action may require a mixture of complex methodologies and considerable time to unravel any causal relationship" (McQueen & Anderson, 2001, p. 77). The meaning of the term multi-strategy, however, varies greatly. For some, multi strategy corresponds to the use of multiple methods and information data that allow for the participative evaluation of multiple dimensions, like outcome, process, and social and political context (Carvalho, Bodstein, Hartz, & Matida, 2004; Pan American Health Organisation, 2003). For others, the support for using multiple methods and strategies is rooted in wills to deploy multi-paradigm designs (Goodstadt et al., 2001). More generally, however, the term refers to studies mixing qualitative and quantitative methods of enquiry (Gendron, 2001; Green & Caracelle, 1997). Exceptionally, in the evaluation literature, multi-strategy also refers to the possibility to mix all kinds of evaluation approaches or models from diverse categories, such as advocacy, responsive, and theory-driven evaluation (Yin, 1994; Datta, 1997a,b; Stufflebeam, 2001). In all these references, the use of multi-strategy evaluation is justified as the best approach to minimize validity problems in dealing with the complexity of multi-strategy interventions and in multi-centers evaluation research. Unfortunately, in examining research synthesis studies it is often impossible to estimate the real utilization and the effective contribution of multi-strategy evaluation, despite the fact that such multi-strategy evaluation is largely recommended to improve knowledge resulting from health promotion intervention evaluations. Meta-analysis and other research synthesis methods are based on a very limited classification system of evaluation study design that consists in whether a Randomized Control Trial (RCT) has been used (Hulscher, Wensing, Grol, Weijden, & Weel, 1999; International Union for Health Promotion and Education, 1999). This impedes the capacity to judge the appropriateness of evaluation approaches, in particular for the multi-strategy interventions characterizing complex communitybased actions. Considering the additional difficulties associated with conceptual definitions of health promotion in community settings (Boutilier, Rajkumar, Poland, Tobin, & Badgley, 2001; Potvin & Richard, 2001) and the absence of a standardized typology for multi-strategy evaluations and their implications to research validity and practical utility, this chapter explores the approaches and multi-strategy models implemented by evaluators in health promotion. To do so, we carried out a systematic review of scientific articles reporting on community health promotion evaluation conducted in countries from any of the three Americas, between 2000 and 2005, and available through electronic databases until May 2005. We were further interested in assessing the quality of these evaluation studies using quality indicators derived from international standards of meta-evaluation adequacy and from health promotion principles and values. Two questions guided our work: (1) What are the characteristics of health promotion intervention evaluation studies? and (2) To what extent do these studies conform to common and specific evaluation standards? The need for using specific standards comes from the fact that, in order to convincingly demonstrate both expected and unintended effects, evaluation must use methodological approaches that are congruent with the principles and values of complex community health promotion interventions. © 2008 Springer Science+Business Media, LLC.