The theoretical and practical difficulties of evaluating a community-based ‘whole systems’ obesity prevention intervention: a research team’s critical reflection

Abstract

Aims: This article critically discusses the purpose, pragmatics and politics of conducting commissioned evaluations on behalf of public sector organisations by drawing on the experience of evaluating a community-based ‘whole systems’ obesity prevention intervention for an English local council. Methods: The study presented in this article incorporated two approaches: an evaluability assessment that interrogated the theoretical and practical difficulties of evaluating the intervention in a non-political way, and a retrospective analysis using Soft Systems Methodology that interrogated the more political difficulties of conducting such an evaluation in the ‘real world’. The information and insights that enabled these reflections came from over 3 years of working closely with the programme team, attending and participating in stakeholder events and meetings, presenting to the Council’s Scrutiny Committee meetings, four interviews with the programme manager, and multiple face-to-face group meetings, email exchanges and telephone conversations. Results: The study reveals and analyses three key inter-related challenges that arose during the evaluation of the ‘whole systems’ obesity prevention intervention: the programme’s evaluability, the evaluation purpose, and the nature, role and quality of evidence. Conclusions: The evaluability assessment was important for defining the programme’s theoretical and practical evaluability, and the retrospective analysis using Soft Systems Methodology enabled a greater understanding of the political tensions that existed. Key learning points related to the challenges that arose during this evaluation have broad applicability.