Developing evaluation questions: Beyond the technical issues

Abstract

This chapter is a contribution to the evaluation debate in health promotion. It explores the strategies for, and main concerns in, identifying and formulating relevant and answerable evaluation questions. Unlike many previous texts discussing what is involved in developing evaluation questions, in this chapter we argue that a wide range of contextual and highly political factors contribute to the framing of evaluation questions. Furthermore, we strongly believe that the same intervention can lend itself to a variety of different evaluation questions depending upon the stage of intervention development at which the evaluation is conducted. Finally, we contend that the evaluation question or set of questions should reflect the ever-changing context of the intervention, as well as the stage if the intervention, if evaluation is to be truly useful for health promotion. The Cochrane Collaboration defined the relationship of health promotion and public health in this way: "health promotion and public health encompasses the assessment of the health of populations formulating policies to prevent or respond to health problems promoting healthy environments and generally promoting health through the organized effort of society. Public health promotes societal action to invest in living conditions that create, maintain and protect health. This covers an extremely wide range of interventions aimed at improving health, with various levels and types of interventions included" (EPPI ' Centre, 2006). Health Promotion expands upon the definition of health by addressing the complexity of social changes, reiterating the importance of acting not only on the issue of demedicalization and reorientation of health services and practices, but especially in the sphere of local development and empowerment.Carvalho, Bodstein,Hartz, and Matida (2004) discussed political and economic determinants of the health-disease process, reaffirming health as an ethical imperative and a citizen's right. There are many elements common to the numerous different definitions of evaluation available in the literature. These include describing, comparing and assessing the value of programs and interventions in the pursuit of specific aims and, increasingly, incorporating lessons learned into the decision-making process (OECD, 1998). Other authors conceive of evaluation as a process of systematic and objective appraisal of a project, program or policy (Shadish, Cook,& Leviton, 1995). According to Hawe, Degeling, Hall, and Brierley (2003) evaluation is a judgement about something. These authors assert that the way in which these judgments are made depends upon expectations, past experiences, and what relevant actors believe to be important. For (Battista et al., 1999), evaluation reinforces the critical link between science and policy, and attempts to reconcile those two worlds, which operate within different paradigms.While Western science often adopts a positivist paradigm that assumes the existence of truth, policy-making is an interpretive process oriented toward the integration of various factors into operating decisions. In addition to assessing interventions' success, evaluations are also oriented toward obtaining information in order to interpret what has happened in interventions, particularly through participatory processes and techniques. They take various forms, including process evaluation, participatory evaluation, formative evaluation, empowerment evaluation, and illuminating evaluation (MacDonald, Veen, & Tones, 1996). Effectiveness evaluation has been highly debated in recent years, and the term is loaded with many connotations. As a particular type of outcome evaluation (Weiss, 1998) effectiveness evaluation is increasingly considered to be an accepted standard for health promotion evaluation (McQueen & Jones, 2007). Issues and criticism regarding effectiveness evaluation are often related to the question: Do we need evidence of effectiveness to make decisions in order to accomplish healthpromotion objectives? Potvin, Haddad, and Frohlich (2001) have discussed the fact that evaluation questions need to reflect a comprehensive understanding of health promotion programs they are intended to address. This paper takes this argument a step further and discusses the wide range of other factors that influence and shape the process by which evaluation questions are identified and formulated. Many issues related to evaluation in health promotion have been raised over the years. Discussions pertaining to the definition of the subjects and objects of the evaluation, the criteria for selecting and developing appropriate evaluation questions; the variables and indicators for measuring and rating success; the various methodological approaches to data collection; and the relationship between evaluators and decision-makers or between policy and research, can all be found in the health promotion literature. In this chapter our aim is to bring together new and old arguments in favor of focusing health promotion evaluation on a variety of relevant aspects of health promotion interventions in order to strengthen health promotion theory and practice. The issue of where to focus the evaluation is often perceived as simply a matter of the stage of the intervention, or decision-makers' needs for information. Developing and formulating an evaluation question goes far beyond the technical aspects of formulating an answerable question.We argue that it is essentially a practical issue; it is a complex process, iterative in nature, which involves negotiation among the various stakeholders. This negotiation is most often political in nature, and requires tradeoffs and compromises on the scope of the evaluation, themethodological approaches required and the political relevance of the information produced by the evaluation. The iterative aspect of identifying and formulating an evaluation question reflects the constantly changing context of the intervention, and consequently, the evaluation. Different questions can be formulated during the life cycle of the healthpromotion intervention, which are influenced by the context, the evolution and the changing nature of the intervention, as well as by the demands of decision-makers or other stakeholders. This point is illustrated in Chapter 12 of this book in the discussion of the challenges of evaluating intersectoral initiatives. © 2008 Springer Science+Business Media, LLC.