��Copyright JASSS Paul Windrum, Giorgio Fagiolo and Alessio Moneta (2007) Empirical Validation of Agent-Based Models: Alternatives and Prospects Journal of Artificial Societies and Social Simulation vol. 10, no. 2, 8 http://jasss.soc.surrey.ac.uk/10/2/8.html For information about citing this article, click here Received: 22-May-2006 Accepted: 08-Jan-2007 Published: 31-Mar-2007 Abstract This paper addresses a set of methodological problems arising in the empirical validation of agent-based (AB) economics models and discusses how these are currently being tackled. These problems are generic for all those engaged in AB modelling, not just economists. The discussion is therefore of direct relevance to JASSS readers. The paper has two objectives. The first objective is the identification of a set of issues that are common to all modellers engaged in empirical validation. This gives rise to a novel taxonomy that captures the relevant dimensions along which AB modellers differ. The second objective is a focused discussion of three alternative methodological approaches being developed in AB economics ��� indirect calibration, the Werker-Brenner approach, and the history-friendly approach ��� and a set of (as yet) unresolved issues for empirical validation that require future research. Keywords: Methodology, Empirical Validation, Agent-Based Models, Simulation, Calibration, History-Friendly Models Introduction 1.1 This paper identifies a set of fundamental validation problems faced by all those engaged in agent-based (AB) economics, and assesses the strengths and weaknesses of alternative empirical validation procedures that have been developed in recent years. The paper locates the problems, and proposed solutions, within 3 domains: (1) the relationship between theory and empirical research, (2) the relationship between models and the real-world systems being modelled, and (3) the way in which a validation procedure deals with (1) and (2). These issues are generic and apply to all those engaged in AB modelling. The discussion in this paper is therefore highly relevant to JASSS readers. 1.2 Before proceeding, let us define what is meant by AB models[1]. These tend to contain the following three ingredients. 1. Bottom-up perspective. The properties of macro-dynamics can only be properly understood as the outcome of micro-dynamics involving basic entities/ agents (cf. Tesfatsion 2002). This contrasts with the top-down nature of traditional neoclassical models, where the bottom level typically comprises a representative individual and is constrained by strong consistency requirements associated with equilibrium and hyper-rationality. Conversely, AB models describe strongly heterogeneous agents living in complex systems that evolve through time (Kirman 1997a 1997b). Therefore, aggregate properties are interpreted as emerging out of repeated interactions among simple entities rather than from the consistency requirements of rationality and equilibrium imposed by the modeller (Dosi and Orsenigo 1994). 2. Boundedly-rational agents. Since the environment in which economic agents interact is too complex for hyper-rationality to be a viable simplifying assumption (Dosi et al. 2005), one can, at most, impute to agents in AB models some local and partial (both in time and space) principles of rationality, e.g. myopic optimisation rules. AB modellers maintain that socio-economic systems are inherently non- stationary, due to persistent novelty (e.g., new patterns of behaviour) endogenously introduced by the

agents themselves. Hence, agents face 'true (Knightian) uncertainty' (Knight 1921) and are only able to partially form expectations, e.g. on technological outcomes. New technologies are introduced into open-ended technological spaces, and payoffs to R&D are non-static and cannot be known ex ante (Nelson and Winter 1982 Dosi 1988). As a consequence, agents face the extremely difficult task of learning and adapting in turbulent, endogenously changing, environments. On this basis, AB researchers argue that assumptions of individual hyper-rationality coupled with rational expectations are inappropriate starting points for modelling. Rather, agents should be assumed to behave as boundedly rational entities with adaptive expectations. 3. Networked direct interactions. Interactions among economic agents in AB models are direct and inherently non-linear (Fagiolo 1998 Windrum and Birchenhall 1998 Silverberg et al. 1988). Agents interact directly because current decisions directly depend, through adaptive expectations, on the past choices made by other agents in the population (i.e. a widespread presence of externalities). These may contain structures, such as subgroups of agents or local networks. In such structures, members of the population are in some sense closer to certain individuals in the socio-economic space than others. These interaction structures may themselves endogenously change over time, since agents can strategically decide with whom to interact according to the expected payoffs. When combined with heterogeneity and bounded rationality, it is likely that aggregation processes are non-trivial and, sometimes, generate the emergence of structurally new objects (Lane 1993a 1993b). 1.3 A number of important consequences follow: First, agents in AB models typically learn by engaging in an open-ended search of dynamically changing environments (Dosi et al 2005). Indeed, agents are not initially endowed with an understanding of the underlying structure of the environment in which they operate, but must develop a representation of its underlying structure. The introduction of endogenous novelty makes the task more difficult since the introduction of new objects alters this underlying structure and, hence, the payoffs associated with alternative actions. Furthermore, the complexity of the interactions between heterogeneous agents underpins open-ended search. Second, partly as a consequence of adaptive expectations, AB models are characterised by true, non- reversible, dynamics, i.e. the state of the system evolves in a path-dependent manner (Marengo and Willinger 1997). The focus is on the self-organizing properties that emerge through these feedback loops. As Silverberg et al. (1988) observe, in economics we see "complex interdependent dynamical systems unfolding in historical, i.e. irreversible, time, economic agents, who make decisions today the correctness of which will only be revealed considerably later, are confronted with irreducible uncertainty and holistic interactions between each other and with aggregate variables" (Silverberg et al. 1988: 1036, italics in original). Third, selection-based market mechanisms are sometimes at work in AB models (see, for instance, the model of technological change pioneered by Nelson and Winter 1982). Most obviously, the goods and services produced by competing firms are selected by consumers. The selection criteria that are used may themselves be complex and span a number of dimensions. Turbulence in industry dynamics can be created through successive rounds of firm entry and exit (Saviotti and Pyka 2004 Windrum and Birchenhall 2005 Windrum 2005). 1.4 AB researchers have enjoyed significant success over the last 20 years. Despite the deep philosophical differences that exist between AB and neoclassical models[2], orthodox (neoclassical) economists have recognised the significance of the AB critique, and have reacted by extending their own modelling framework to incorporate (certain) aspects of heterogeneity, bounded rationality, learning, increasing returns, and technological change. Yet orthodox economists have not been moved to join the AB camp. There are many possible explanations for this but an important aspect, recognised by AB modellers themselves, is a perceived lack of robustness in AB modelling. This threatens the AB research enterprise as a whole. Four key problem areas were identified in a recent conference and special workshop attended by the authors[3]. First, the neoclassical community has consistently developed a core set of theoretical models and applied these to a range of research areas. The AB community has not done this. Indeed, the sheer diversity of alternative AB models put forward over the last 20 years is striking. 1.5 A second, related set of issues concerns a lack of comparability between the models that have been developed. Not only do the models have different theoretical content but they seek to explain strikingly different phenomena. Where they do seek to explain similar phenomena, little or no in-depth research has been undertaken to compare and evaluate their relative explanatory performance. Rather, models are viewed in isolation from one another, and validation involves examining the extent to which the output traces generated by a particular model approximates one or more 'stylised facts' drawn from empirical research. 1.6 This leads us to a third issue: the lack of standard techniques for constructing and analysing AB models. It has been argued that developing a set of commonly accepted protocols for AB model building would benefit

the profession (Leombruni 2002 Richiardi 2003). This would address, for instance, issues such as how and when sensitivity analysis (over the space of initial conditions and parameters) should be conducted, how one should deal with non-ergodicity in underlying stochastic processes, and how one should interpret, in terms of real-world time, the timing and lag structures that AB modellers typically build into their models. 1.7 Finally, the fourth set of issues concerns the problematic relationship between AB models and empirical data. Empirical validation of a model (M) involves an assessment of the extent to which M is a good representation of the (unknown) process that generated a set of observed data. This opens up a raft of fundamental questions. What is the methodological basis that informs the process of empirical validation? Is this process specific to AB models, or is it generic to all modelling enterprises? A central objective of this paper is the exploration of these issues. 1.8 As well as there being great diversity in the way AB modellers go about developing models, fundamental differences exist in the way they conduct empirical validation. Key areas of debate include whether a 'realist' methodology is appropriate? Why should empirical validation be the primary basis for accepting or rejecting a model? Do other forms of model validation exist besides the reproduction of stylised facts? If we do proceed down the path of empirical validation, then how should one relate and calibrate the construction of parameters, initial conditions, and stochastic variability in AB models to the existing empirical data? Which classes of empirically observed objects do we actually want to replicate? How dependable are the micro and macro stylised facts that are to be replicated? To what extent can we truly consider output traces to be stylised facts or, alternatively, counterfactuals? What are the consequences, for the explanative power of a model, if the stylised facts are actually 'unconditional objects' that only indicate properties of stationary distributions and, hence, do not provide information on the dynamics of the stochastic processes that generated them? Given these rather fundamental questions, it should not come as a surprise that very different approaches to empirical validation are found in the AB literature. A novel contribution of this paper is its explicit consideration of the basis for this heterogeneity. 1.9 The paper is structured as follows. Section 2 discusses the methodological basis of empirical validation, i.e. the comparison of discrete-time models with empirical data. The section begins with a discussion of the core issues of empirical validation faced by all modellers (neoclassical as well as AB economists). Having identified these core issues, section 3 opens the discussion on methodological diversity within the AB community. We suggest that this methodological heterogeneity is due to two factors. The first factor is the problem of modelling highly non-linear systems, i.e. systems with stochastic dynamics, non-trivial interactions among agents, and feedbacks from the micro to the macro level. The second factor is the diverse structural content of AB models, and the very different ways in which AB models are currently analysed. We present a novel taxonomy of AB models that captures the salient dimensions of this diversity. There are four dimensions: (1) the nature of the object under study, (2) the goal of the analysis, (3) the nature of the main modelling assumptions, and (4) the method of sensitivity analysis. 1.10 Building on section 3, section 4 provides a detailed survey of three major approaches to AB empirical validation. These are the indirect calibration approach (4.4), the Werker-Brenner approach to empirical calibration (4.10), and the history-friendly approach (4.18). We examine the strengths and weaknesses of each approach. This paves the way for a discussion, in section 5, of a set of unresolved issues for empirically- oriented AB modellers. Section 6 concludes. The methodological bases of empirical validation: comparing discrete-time models with empirical data 2.1 In this section we clarify the process of empirical validation and identify a set of validation issues that are common to all modellers (neoclassical and AB alike). Let us consider the typical situation faced by any empirically-grounded economist attempting to replicate and/or explain a set of stylised facts. The point of departure is almost always a set of empirically observed data (e.g. panel data) whose generic form is: (z)i = { zi, t, t = t 0, ���, t1}, i ��� I. Here the set I refers to a population of agents (e.g. firms and households) whose behaviour has been observed across the finite set of time-periods {t0, ���, t1} and refers to a list of, say, K variables contained in the vector z. Whenever agent-level observations are not available, the modeller has access to the K-vector of aggregate time-series: Z = { Zt, t = t 0, ���, 1},