How to evaluate a monitoring system for adaptive policies: criteria for signposts selection and their model-based evaluation

Abstract

Adaptive policies have emerged as a valuable strategy for dealing with uncertainties by recognising the capacity of systems to adapt over time to new circumstances and surprises. The efficacy of adaptive policies hinges on detecting on-going change and ensuring that actions are indeed taken if and when necessary. This is operationalised by including a monitoring system composed of signposts and triggers in the design of the plan. A well-designed monitoring system is indispensable for the effective implementation of adaptive policies. Despite the importance of monitoring for adaptive policies, the present literature has not considered criteria enabling the a-priori evaluation of the efficacy of signposts. In this paper, we introduce criteria for the evaluation of individual signposts and the monitoring system as a whole. These criteria are relevance, observability, completeness, and parsimony. These criteria are intended to enhance the capacity to detect the need for adaptation in the presence of noisy and ambiguous observations of the real system. The criteria are identified from an analysis of the information chain, from system observations to policy success, focusing on how data becomes information. We illustrate how models, in particular, the combined use of stochastic and exploratory modelling can be used to assess individual signposts, and the whole monitoring system according to these criteria. This analysis provides significant insight into critical factors that may hinder learning from data. The proposed criteria are demonstrated using a hypothetical case, in which a monitoring system for a flood protection policy in the Niger River is designed and tested.