Empirical Calibration of Simulation Models

Claudia Werker and Thomas Brenner














Eindhoven Centre for Innovation Studies, The Netherlands
Working Paper 04.13



Department of Technology Management
Technische Universiteit Eindhoven, The Netherlands

June 2004

EMPIRICAL CALIBRATION OF SIMULATION MODELS

Claudia Werker and Thomas Brenner*






Abstract

This paper discusses how the results of simulation models can be made more reliable
and the method of simulating therefore more widely applicable. We suggested to
calibrate simulation models empirically and developed a methodology based on Critical
Realism in order to so. We suggested combining the procedures of two strands of
literature: the empirical underpinning of the assumptions (like in microsimulations) and
the empirical check of the implications (like in Bayesian inference). Both these strands
of literature are mainly concerned with predicting future developments. We, instead, aim
to infer statements about causal relations and characteristics of a set of systems or
dynamics, such as, e.g., the development of an industry, that have a general validity for
this set of systems or dynamics. In other words, instead of deriving probabilistic
predictions of the future and statements of the current situation and dynamics of one
single system we developed a methodology to gain general statements about the features
of systems and dynamics.






*Claudia Werker, Eindhoven University of Technology, ECIS, email: c.werker@tm.tue.nl and Thomas
Brenner, Max Planck Institute for Research into Economic Systems, Jena, email: brenner@mpiew-
jena.mpg.de.

Table of Content

1. Introduction ....................................................................................................................1
2. Modelling the Real World..............................................................................................3
2.1 Elements of Models............................................................................................. 3
2.2 Inference in Modelling ........................................................................................ 5
3. Critical Realism as the Methodology for the Empirical Calibration of Simulation
Models................................................................................................................................8
3.1 Empirics in Simulation Models........................................................................... 8
3.2 Positivism and Critical Realism as Methodologies in Economic Modelling.... 10
3.3 Critical Realism in Simulation Models ............................................................. 12
4. Inference in Empirically Calibrating Simulation Models ............................................13
4.1 Induction and Deduction of the Set of Assumptions......................................... 13
4.2 Deduction and Induction of Implications.......................................................... 14
4.3 Abduction of a Set of Models ........................................................................... 17
5. Conclusions ..................................................................................................................19

1
1. Introduction

In economics simulation models are used quite a lot to carry out mathematical
experiments. However, the specification of the parameter set with which to run these
simulations is, in general, quite an adventure into the unknown. Criticism is easily
found with the procedure, as it is difficult to justify why to choose one specification of
parameters and not another - especially if the results found in the simulation models
are striking. Then, the audience cannot help but think that there has been quite some
arbitrary trial and error going on to achieve this. To avoid this impression, we suggest
to empirically calibrating simulation models in a way that makes their results more
acceptable. However, compared with models of mainstream economics that usually
can be solved analytically, simulation models have only recently been opened to
empirical data. It is fair to say that also analytically solvable models have quite some
problems concerning the integration of empirical data (cf. Kydland/Prescott, 1996).
These problems also emerge when working with simulation models.

We will suggest two ways of underpinning the empirical calibration of simulation
models methodologically. First, we suggest combining the empirical underpinning of
the assumptions with the empirical check of the implications. To this end, we build on
two strands of literature in simulation modelling. The one strand is concerned with
microsimulations, where empirical knowledge is used to set up the simulation model
(cf, e.g., Citro & Hanushek 1991). The other one is the statistical approach under the
label of Bayesian inference, where empirical data is used to test the simulation model
(cf. Zellner, 1971). Using different two different data sets in this way to confront
assumptions as well as implications with reality would also help to improve the way
every model deals with empirical data and is not restricted to simulation models.

The second way we suggest underpinning the empirical calibration of simulation
models methodologically lies in the very nature of simulation models. In contrast to
analytically solvable models simulation models face a severe problem and that is that

2
uncertainty is at their very heart.1 While most analytically treated models in the
economic literature describe deterministic processes, most simulation models deal
with stochastic processes. Therefore, the solutions are contingent, i.e. subject to a
combination of chance and necessity. How to deal with chance and necessity within
simulation models is crucial when modelling both aspects that are of course
intertwined. It becomes even more difficult to deal with these two aspects when
simulation models have to be empirically calibrated, because historical events take
only place once and one has to identify the characteristics a number of historical
events have in common and the characteristics that emerge from chance in the data.

We suggest using Critical Realism as methodology, because it helps to categorize
empirical events actually taking place and to determine the underlying structural
driving forces. We will show that this approach is the most promising way to use
simulation models for inferring general knowledge about the features of a set of
systems or dynamics. By this simulations become a more interesting and reliable tool
for understanding economic processes and developments. To show how empirical
data can be used to make simulation models more widely acceptable and applicable,
we first look into what one has to take into consideration when modelling the real
world by way of simulation models (Section 2.). Then we show particularly how the
methodological approaches of Positivism and Critical Realism can be used for
economic modelling (Section 3.). Based on this discussion we explore into the
question of how Critical Realism can serve to examine the features of economic
processes with the help of empirically founded simulation models (Section 4.). We
conclude with a brief summary and an overview of additional questions one would
like to answer in the context of the empirical calibration of simulation models
(Section 5.).


1 Please note that we do not consider mainstream models that use simulations as a mathematical tool to
solve underdetermined equation systems as simulation models here. For a more detailed discussion of
the kind of simulation models we consider see Section 3.1.

3
2. Modelling the Real World
2.1 Elements of Models

To model the real world, theories use different elements and abstract from what is
actually going on in the part of reality they want to describe, explain, or prognosticate.
Sometimes the term “model” is defined as being a “theory” that is expressed in
equations. As this leads to a couple of questions that are not interesting in the context
of our analysis,2 we use the terms “model” and “theory” synonymically here. The
most important elements of models are premises, definitions, logical sentences, causal
relationships3 as well as data. Every model starts from premises that limit the area of
application of the model, e.g. concerning time, place, agents involved etc. Not all
premises are made explicitly. One famous premise, that is often not even mentioned,
because everybody is expected to know that it applies, is the “ceteris-paribus-clause”.
Definitions are conventions about how to name elements of reality. They are not true
or wrong. They simply help to communicate ideas. Not all definitions are formulated
explicitly. Usually the exogenous and endogenous variables as well as parameters that
are relevant in the theory are defined. However, definitions of terms, with which
everybody in the field is familiar, are often not given.

Logical sentences are at the very heart of putting together models, because they
combine complex and complicated relationships in a consistent way. Axioms are
important logical sentences, which normally can be expressed in mathematical terms.
Another important kind of logical sentences are causal relationships, which give
information about cause and effect. Often researchers formulate causal relationships
in the form “if … then …”. In case a researcher wishes to explain an economic
situation the “then …” part of the causal relationship is known whereas the “if …”
part, i.e. the cause is searched for (cf. Machlup, 1978, 455f). In case a researcher
wishes to prognosticate a future economic situation it is the other way around, i.e. the

2 E.g. one question would be: Is it sufficient that a theory can be potentially expressed in equations to
turn it into a model?
3 Causal relationships are also often co notated as hypotheses.

4
cause is known and the effect is searched for. Therefore, causal relationships can say
something about the functioning of the real world in the past and the future.

Data is particularly important in our further discussions as it contains claims about
parts of reality, which play an important role in inference (see Section 2.2.). When
discussing how to derive data it is crucial to be aware that "... (e)mpirical analysis in
any research field is entwined in theoretical analysis. That is, empirical work depends
on theory for concepts, definitions and hypotheses, all of which are used as
foundations for empirical investigation" (Coward/Foray, 2002, p. 540). This means,
that we do not only use data to build our theories and to check their implications but
also that we use theory to produce data from the complex and complicated processes
going on in reality. Consequently, a number of problems emerge from data collection.
Collecting data requires making a couple of choices and theorizing about how to
observe and measure (cf. the following Machlup, 1978, 448-450). When researchers
collect the data themselves they can make these choices. Often researchers rely on
data collected by others, which means that aspects important for their research
questions might not be taken into consideration (sufficiently). However, even if
researchers collect the data themselves it might be difficult to observe the relevant
aspects. There might emerge some measurement problems. Moreover, data is usually
not available for everything within the area of application defined by the premises.4
This leads to the well-known problem of induction that even if you observe a large
part of reality there is no possibility to make all observations – in particular not those
in the future. Insofar, it is impossible to verify a model, as there might always be
evidence to the contrary.

Data can be obtained through one detailed study, which is open to critical
consideration as in such an analysis the problems connected with the production of
data become obvious. Usually data found in this way is not covering a wide
application area. Therefore, sometimes so-called stylised facts are used following
Kaldor's suggestion (Kaldor, 1968, 177f). Stylised facts comprise statements about a
wide application area. The problem with stylised facts is that they fall from heaven

4 For an analysis of measurement of technical advance as well as problems connected with this see
Grupp, 1998.

5
and often remain unmotivated. In order to keep the broad application area but to avoid
the pitfalls of stylised facts the concept of structural regularities was developed, which
modifies Kaldor’s concept. In contrast to Kaldor’s original – and still widely used –
approach, the concept of structural regularities is based on strict guidelines for the
identification of these regularities (Schwerin, 2001, 92-117). Moreover, it explicitly
considers chance and necessity elements in economic processes and helps to
distinguish the both. The disadvantage of structural regularities as an empirical basis
for theories lies in the fact that it requires a lot of work done on the data part (e.g.
Schwerin/Werker, 2003).

2.2 Inference in Modelling

In general, models can be distinguished into two major parts: assumptions and
implications. Each element of models, e.g. data or premises, can be part of the
assumptions and/or of the implications. Where you find them depends on the principle
of inference or the combination of principles of inference used. Three different
principles of inference can be distinguished: deduction, induction and abduction.
Premises and definitions are usually part of the assumptions as these elements set the
boundaries for modelling. However, sometimes definitions and premises can also be
part of the implications, especially so if the results of a model indicate that premises
and/or definitions have to be revised for further research. Data can be used in both
parts of models. In assumptions data provides an empirical basis to start from. In
implications data is used to corrobate implications stemming from premises,
definitions and logical considerations. Logic is of course always at the heart of
modelling in all parts and puts all elements of the models together in a consistent way.

In the following, we will show how the three different inference principles work. It is
important to notice that modelling usually combine different principles of inference,
thereby coming from assumptions to implications. Each principle of inference works
in the different way, although meeting the same end, namely inferring implications
from assumptions.

6
Deduction is often summarized as inferring “from general to particular” (cf. Lawson,
1997, 24). From assumptions implications are derived in an analytical and logical
way. For example you infer from the general assumption “all ravens are black” the
particular implication “that the next raven you observe will be black as well”.
Assumptions within deduction already contain all information that there is available.
Generally spoken, deduction sustains the information contained already in the
assumptions but does not create new one.

If A = B and B = C, (assumptions)
then A = C. (implication)

In deduction assumptions contain all possible elements of models, like e.g. premises,
definitions or causal relationship. Therefore, it is often claimed that inference in
deduction is necessary in the sense that the conclusions stemming from the
assumptions are correct. In formal sciences like mathematics this holds, because
assumptions are often provided in the form of axioms, i.e. they are self-evident and
need not be proven. In social sciences like economics such self-evident assumptions
do not exist. Implications drawn from premises are in general true but only in the
sense that they are logically derived. In social sciences without self-evident premises
available it is virtually impossible to derive implications that are true in the sense of
correctly describing, explaining and prognosticating reality.

Induction is often summarized as inferring “from particular to general” (cf. Lawson,
1997, 24). Its assumptions describe a part of a larger population and then infer
conclusions about the characteristics of this larger population. For example you start
from the assumption of the particular observation “that you saw numerous black
ravens today” and infer from this the general implication “that all ravens are black
nowadays”. As the inductive principle runs “from particular to general” it is often
considered as creating information - however doubtful one. The inference in induction
says something not contained in the assumptions. If the inference arguments are
strong it is probable that the claims made about the conclusions hold. Inductive
inference is based on data. However, even if the number of observations in the data
set is huge it is in principle impossible to have all observations available, not the least
because future events cannot be observed. This means that the implications derived

7
from data are uncertain. In the future, the same will only happen with an unknown
probability. This probability is impossible to gain, because future observations can by
definition not be made.

Abduction5 classifies “particular events into general patterns” (Lawson, 1997, 24).
You start, e.g., from the particular observation “that there are numerous black ravens”
and try to undercover the underlying mechanisms about what is “disposing ravens to
be black”. You might, e.g., look for other animals that are black and study what they
have in common, or look for the differences between ravens and other birds that are
not black. It is important to notice that abduction requires data based on substantial
and detailed observations. Only then is it possible to find meaningful and sensible
underlying mechanisms to infer from the assumptions to the implications. So, e.g., if
we observe that ICT firms are usually having staff members below 40 years old and
we see a particular person who is less than 40 years old we might conclude that this
person is an ICT employee. Obviously, this is quite jumping to conclusions.
Abduction requires much more detailed information to infer implications that are
likely to hold when confronted with reality. In our example one would wish to know
much more about the background of ICT employees, e.g. their education etc. to
identify a general pattern. Then, it would be possible to classify a person according to
these characteristics and to conclude whether or not s/he is an ICT employee.
Whereas an age of less than 40 years is the characteristic of very many people a
combination of this characteristic with a degree in informatics would make it much
more likely that the person works in the ICT sector. The more relevant details are
known about the data the more precisely they can be classified to a general pattern.

Abduction enables us to identify underlying structural elements, which explain
observations we make, and to develop a theory of the part of the world we are
investigating. This takes us a substantial step further than pure deduction or induction,
because abduction helps us to meet theory and data in a creative way. By using the
principle of abduction we are able to create new information. According to Peirce
(1867, Vol. 5, 145):


5 Abduction is often also called retroduction.

8
“(Induction) never can originate any idea whatever. No more can deduction. All the ideas of science
come to it by the way of abduction. Abduction consists in studying the facts and devising a theory to
explain them. Its only justification is that if we are ever to understand things at all, it must be in this
way.”

3. Critical Realism as the Methodology for the Empirical Calibration of
Simulation Models

We want to show how simulation models can be empirically calibrated and which
methodological principles have to be followed to achieve this in an appropriate and
meaningful way. From the methodologies used to develop economic models we will
look into Positivism and Critical Realism. To discuss Positivism is important, as it is
the mostly used methodological basis for economic modelling. In contrast, Critical
Realism is used only rarely. However, we will show that this methodology is best
suited to empirically calibrate simulation models. First, the role empirics played in
simulation models so far is discussed (Section 3.1). Then, the features of Positivism
and Critical Realism are analysed to show that Critical Realism is best suited to meet
the requirements of calibrating simulation models (Section 3.2).
3.1 Empirics in Simulation Models

Simulation models in our understanding are tools of heterodox economics and
therefore are normally used in contexts where agents are heterogeneous and decide
under uncertainty. These features usually lead to underdetermined equation systems
so that simulation methods have to be used to find (range of) solutions for the models.
However, it is important to note that we do not consider mainstream models that use
simulations as a mathematical tool to solve underdetermined equation systems as
simulation models here. In these models, assumptions about the behaviour of
economic agents, the structure of the economic systems, and the underlying
mechanisms are made and the implications calculated. The only difference is that
simulating the results allows for more complex models but causes on the other hand
more difficulties in analysing the results. Such an approach still faces the problem that
in social sciences no self-evident axioms exist. Nevertheless, protagonists of such an

9
approach do as if there are such self-evident axioms, from which necessary
implications can be derived.

There are only few simulation models of the heterodox kind that include empirical
elements, i.e. history-friendly models (Malerba, Nelson, Orsenigo & Winter 1999 and
Malerba & Orsenigo 2002) and microsimulations (Citro & Hanushek 1991). History-
friendly models represent case studies and lack the possibility of generalization. They
start with some empirical knowledge about real processes – in the one case some
stylised facts and in the other case detailed knowledge about one historical realisation
– and try to find a model that leads to processes with the same characteristics.
Although this is not mentioned in the final publications, this means that different
models are tested and rejected by the empirical knowledge until a model is found that
is not rejected. It is then argued that the model might describe the mechanisms
underlying the known empirical facts. Hence, an inference is made from some, often a
few empirical facts to a model describing the whole complex of involved processes.
The same holds for many mathematical models in mainstream economics, although
this topic is less discussed there. This kind of simulation approach is often criticised
because of its lack of completeness, its questionable general validity and its lack of
rigour. Nevertheless, such an approach has several advantages in comparison to other
approaches. In contrast to mathematical modelling, such an approach can deal with
complex mechanisms that include random events. Furthermore, through simulations
data can be produced that can then be analysed. This means that a simulation
approach can deal with situations where little empirical data is available and that
mechanisms and relationships can be studied that cannot be directly observed in
reality.

In microsimulations comprehensive empirical knowledge about the changes of a
system in the past is used to model future developments. Typically transition
probabilities and trends in variables and parameters are empirical estimated. These
are, then, used to predict the dynamics in the future. An example is the prediction of
the impacts of policy proposals, such as larger benefits for the disabled (e.g., Bagley,
Burpee & Jetté 2000). Immigration, emigration and birth rates are estimated on the
basis of empirical data about the past whereby trends in these rates are considered.
Then a simulation model is developed that includes these empirically estimated rates

10
and processes. Finally, the developed model is used to make prediction about future
developments. Microsimulations are, in general, used if comprehensive data is
available about the processes on the micro-level. They represent a special case of the
method that we propose below.

3.2 Positivism and Critical Realism as Methodologies in Economic Modelling

From the methodologies used to develop economic models we will look into the
approaches of Positivism and of Critical Realism. By and large economics uses
Positivism as methodological basis for modelling, whereas it uses Critical Realism
only rarely. Positivists combine induction and deduction as principles of inference.
They start from general assumptions and infer implications for economic processes
from them. If positivists include data in their modeling, they confront the implications
from deduction with inductively found results. Their aim is to objectively measure
and quantify observable facts as well as to search for empirical regularities that help
to describe, explain and predict reality. Positivists “… have a notion of causality and
connectedness in their theorising, though make closure assumptions. Two forms of
closure are central to this perspective. The intrinsic condition of closure - which can
be characterised loosely as implying that a cause always produces the same effect ...
The extrinsic condition of closure - which loosely can be understood as implying that
an effects always has the same cause ...” (Downward/Finch/Ramsey, 2002, 482).

Positivism has two problems that are particularly important for our discussion of how
to empirically calibrate simulation models. First of all, it is impossible – like already
stated above – to find axioms in social sciences that hold in general. This limits the
value deduction has for theoretical work in social sciences in general and in
economics in particular. To make statements on such deductively inferred
implications is already doubtful. It does only partly improve the model to then extent
the analysis by confronting the inferred implication with empirical data in an
inductive way. The problem that it is impossible in social sciences to infer
theoretically the initial axioms remains. The second problem stems from the fact that
we want to develop a methodological basis for simulation models used in the
heterodox economics. This means that agents in these models are heterogeneous and

11
decide under uncertainty. These assumptions lead to complex and complicated
patterns of the economic processes to be described, explained and prognosticated.
These patterns cannot be covered by the aforementioned conditions of closure, which
suggest that one cause has one effect and the other way around.

Critical Realism, which we will suggest as the appropriate methodological basis for
simulation models used in heterodox economics, uses abduction as principle of
inference and uses so-called semi-closure to account for the fact that different reasons
can have the same effect and the other way around. Protagonists of this school of
thought recognise that the world is structured into different layers
(Downward/Finch/Ramsey, 2002). They aim at describing and explaining empirical
facts in terms of their underlying structures, i.e. in terms of other layers of reality.
This approach uses abduction to infer from empirical facts and observations to the
general patterns underlying them, thereby giving a causal explanation on a deeper
level. Critical Realists point out that institutions co-evolve with agents own mental
models, thereby providing a situation of quasi-closure, i.e. institutions provide stable
conditions upon which agents can base their behaviour for a certain period of time
(Downward/Finch/Ramsey, 2002, 481f). This means that a specific connection
between cause and effect might remain for a while but also changes over time
(Downward/Finch/Ramsey, 2002, 495).

Simulation models in our understanding (see Section 3.1) are tools of heterodox
economics and therefore are normally used in contexts where agents are
heterogeneous and decide under uncertainty. This means that they face a much more
complex and complicated environment in which they nevertheless have to take
decisions and act. One could jump to the conclusion that under such circumstances it
is impossible to develop any models based on empirical data. However, this is not so.
Protagonists of Critical Realism have started to develop a methodology that is
providing a way to deal with these issues and we will use and further develop their
insights in order to provide a methodological basis for the empirical calibration of
simulation models.

12
3.3 Critical Realism in Simulation Models

In line with Critical Realism, we argue that what we observe in reality is the result of
processes on a deeper level. Therefore, it is not sufficient to describe the relationships
on the observed level. We need to understand these relationships on the basis of the
processes of the underlying level. In the following we will show how the
methodology of Critical Realism can be used to calibrate simulation models in
practical terms.

Although our suggestion contains as the major inference principle to put together
theory and empirics abduction this does not mean that the other principles of
inference, i.e. induction and deduction, are not used. In fact, they are used quite
substantially to prepare the final abductive step. First, we will show how the set of
assumptions is put together by induction and deduction (Section 4.1). We suggest
including empirical data available on the assumptions. Based on that, implications are
inferred by deduction and induction (Section 4.2). Here, empirical data about
implications inferred from the dynamics of the described economic system is used.
The two kinds of data that are used have to be different, because they concern
different levels of the whole system.

In a third and final step, abduction helps us to produce classes of models, which
combine assumptions and implications based on empirical findings, i.e. only those
models are included which are not rejected by confronting either their assumptions or
their implications with reality (Section 4.3). Notice that we do not aim to find one
simulation model that describes reality. We believe that this is impossible. As in
statistics, all that can be done with the help of empirical data are two things. First, we
can reject some models meaning that we restrict the parameters of the general model
to certain ranges, so that only a certain subset of all model specifications is
considered. Second, in a later step we will study the correctness of these
specifications with the help of empirical data on implications (see below).

13
4. Inference in Empirically Calibrating Simulation Models
4.1 Induction and Deduction of the Set of Assumptions

The first step, as usual, is setting the assumptions by defining the system that the
simulation model is intended to describe. This means that the relevant factors and
variables have to be chosen and their interaction has to be built into the structure of
the simulation model. This is usually done according to theoretical consideration and
common knowledge. However, we argue here that the details of the model, the
specification of relations and especially the choice of parameters should be fixed
according to empirical data, i.e. inductively. This is rarely done in the field of
computational and evolutionary economics (some exceptions can be found in Eliasson
& Taymaz 2000, Richards 2002, Brenner and Murmann 2003 and Brenner 2004, Ch.
4). We argue that more can be reached by using simulations. To this end, the premises
on which the model is build should be induced from empirical data whenever this is
possible. Of course, the conceptualisation of variables and parameters can never be
theory-free. However, it is important to base all central assumptions of the model on
empirical knowledge.

Whenever no sufficient data is available or whenever the model should capture
different kinds of systems, the model should be defined as general as necessary. In
such a case ranges should be defined for the respective parameters of the model. If the
data does not allow for determining between different forms of relationships between
the variables of the model, all of them should be included in the model with the help
of additional parameters. The ranges of parameters have to be chosen such that the
modeller is sure that the real values lie within these ranges. Logical sentences and
premises that restrict the area of application of the model can be used to reduce the
ranges of the parameters. However, it has to be made clear how this reduction is
reached.

Hence, we argue that parameters should not be fixed to one value, except if the
empirical data allows for such a fixing. This means that we do not aim for developing
one specific simulation model that reflects one bundle of assumptions. Instead, we go
for a set of simulation models of which each represents one bundles of assumptions.

14
Each specific simulation model – in the following we use for simplicity the term
`model specification´ -- represent one specific choice of parameters and premises (see
Figure 1).










Set of model
specifications
s
One model
specification

Figure 1: Set of model specifications

4.2 Deduction and Induction of Implications

Each model specification can be run separately. This is the usual approach in the
literature, where mainly one specification (with respect to the parameters) of the
simulation model is run and its characteristics are studied. Due to the existence of
stochastic processes in the models, many runs are necessary to obtain a complete
picture of all possible implications of each model specification. Whenever a
simulation is run for one model specification, a certain development of the artificial
system results. We call this a theoretical realisation. Rerunning the simulation for the
same model specification might lead to exactly the same theoretical realisation.
However, because of the stochastic processes that are included in the model, it is more
likely that a different theoretical realisation results. If one model specification is
simulated many times, a set of theoretical realisations results. For each model
specification we can determine such a set of theoretical realisations.

There are an infinite number of model specifications. Therefore, not every model
specification can be studied. A Monte-Carlo approach is chosen. This means that
many model specifications have to be randomly picked and the set of theoretical

15
realisations (depicted as an ellipse in Figure 2) for each of the picked model
specifications have to be studied by deduction. The more model specifications are
examined the higher is the validity of the obtained results. Therefore, a high number
of simulation runs is required for the procedure that is proposed here. However, with
increasing computer power this will become less of a problem in the future.


Set of model
specifications
One model
specification
Space of
realisations
Set of theoretical
realizations for one
model specification












Figure 2: Set of model specifications and sets of realisations

Notice that the random choice of model specifications has nothing to do with the
chance elements that are included in the models. Examining only a (high) number of
randomly picked model specification is simply a device to deal with the problem that
simulations cannot be run for an infinite number of model specifications. This is the
only disadvantage of this method compared to a mathematical analysis of models.
This disadvantage becomes the smaller the larger the number of analysed model
specifications.

The stochastic elements in the models are responsible for the fact that one model
specification can cause different theoretical realisations. As a consequence, two
different model specifications might cause the same theoretical realisation (see the
overlapping ellipses in Figure 2). This is what critical realists mean when they state
that different causes can have the same effect.

16
Now the second kind of empirical data is used, those about empirical realisations of
the dynamics of the whole system that is modelled. The simulation models that are
considered here describe the dynamics of a system that is part of the whole economy.
Usually it will not describe only one specific system but a number of systems that
share common features (e.g. the same processes in different countries or industries).
Whenever one such system and its dynamics are observed, we call this one empirical
realisation of the class of systems that our models aim to represent. Usually it will be
possible to gather data about the characteristics of several empirical realisations.

Let us consider the treatment of one empirical realisation. We can examine for each
model specification whether the observed realisation falls into the range of theoretical
realisations that this model specification predicts. According to the above statements,
there is not only one model specification that is able to predict the empirical
realisation. However, we can reject a number of model specifications on the basis of
the empirical observations. Hence, for each model specification we can statistically
state whether or not it is rejected by the empirical data about one specific realisation
of the system’s dynamics. A subset of model specifications that are not rejected
remains.

Furthermore, for all model specifications that are not rejected by the empirical data
the likelihood for their validity can be given. A Bayesian approach can be used to do
this (see Zellner 1971 and Ghosh & Ramamoorthi 2003). Hence, a probability
distribution over the set of model specifications is obtained for each realisation that is
empirically observed.

In the literature such an approach is taken in statistics under the name of Bayesian
inference (see Zellner 1971). There a set of models is defined and for each model the
likelihood of its adequateness is empirically estimated. Each model is then weighted
by its likelihood and predictions are made on the basis of the weighted sum of the
predictions of each model. This can also been done in the approach taken here. The
major difference is that the method proposed here uses empirical data extensively also
for the development of the set of models that are tested. Furthermore, we propose two
further steps in the analysis that are discussed in the next section.

17
4.3 Abduction of a Set of Models

Above we have determined all model specifications that are in line with the
observation of one empirical realisation of the dynamics of the system (ellipse in set
of assumptions in Figure 3), such as, for example, the development of an industry in
one country. Hence, the above procedure allows for obtaining a subset of model
specifications for each empirical realisation. If a number of realisations are observed,
for each of them the subset of model specifications that cannot be rejected can be
determined. Now these subsets can be used to determine the characteristics of the
system. This is done in two steps, which are explained in the following.


Set of model
specifications
Set of model
specifications that are
not rejected by the
empirical realisation
Space of
realisations
One empirical
realisation
Abduction













Figure 3: Abduction Between Set of model specifications and Set of realisations

Usually not only one realisation is observed. In general we look for models that can
explain a number of similar systems (e.g. the developments in different industries or
in different countries). For each single empirical realisation the above method leads to
a subset of model specifications that are in line with this realisation (see the shade
ellipse in Figure 3). If we have a number of empirical realisations, a number of
subsets of model specifications result.

18

It is now possible to classify the empirical realisations in groups, either according to
empirical characteristics (e.g. high-tech and low-tech industries) or according to the
similarity of the subsets of model specifications, which are not rejected through the
above procedure. This means that we define kinds of systems for which we are
interested in their common features. This is the major aim of abduction: a
classification of events, facts or processes and a determination of the characteristics of
each class. Here, it means that we want to define a class of systems or dynamics and
study the characteristics that all systems or dynamics in this class have in common.
Two steps are necessary. First, the class of systems or dynamics that is to be studied
has to be defined, all empirical data, meaning all observations of such systems or
dynamics, has to be gathered and the above method has to be used to identify the
respective set of model specifications (depicted by the arrows leading upwards in
Figure 3). Second, the common characteristics of all realisations that can be obtained
by any model specification in this set have to be studied (depicted by the arrow
leading downwards in Figure 3). Let us assume that a group of empirical realisations
is defined that belongs to the class of systems that is to be studied. The choice of
empirical realisations defines a set of model specifications that consists of those
model specifications that are not rejected by all empirical realisations in the group that
is considered. All these model specifications have to be considered because the aim is
to identify the characteristics that are common to all systems in the chosen group.
Notice that in this case it is impossible to assign a probability to each model. The
empirical realisations represent just a few examples of the dynamics that might be
caused by the studied kind of system. It is in no way clear how we should weight the
induced knowledge of each empirical realisation. This would only be possible if we
have a very large number of observations. Hence, we refer from calculating
probabilities for model specifications and only determine a subset of model
specifications that is in line with the observed realisations (all the area within any of
the ellipses in the upper part of Figure 3).

To understand the characteristics of a class of systems, now all model specifications
that belong to this subset can be simulated. For each model the theoretical realisations
that it might imply can be studied. What kind of characteristics of these realisations is
studied depends on the research question. Everything is possible that is also done in

19
the common simulation approaches that are based on theoretical models. For example,
it is possible to study causal relations or the outcomes of the modelled processes. In
the contrast to the common approaches the model specifications that are used here are
based on an extensive use of empirical data that causes a high validity of the obtained
results. All implications that the whole group of models share are characteristics of
the studied class of systems.

This means that, instead of arguing that there is one model that explains all systems
within a certain class, we argue that a subset of model specifications can be obtained
by abduction. This subset of model specifications contains all possible bundles of
assumptions that cannot be rejected by the empirical data about the systems that are to
be studied. If the model specifications in this subset share characteristics, these
characteristics can be expected to hold also for the real systems (given the
development of the model has not included any crucial and false premise). Hence, we
obtain robust knowledge about the characteristics of a certain kind of systems.

If the characteristics within a group of model specifications differ, the causes of these
differences can be studied. It can be examined which factors in the models are
responsible for the differences. Hence, although we will not know the characteristics
of the real systems in this case, we will obtain knowledge about which factors cause
different characteristics.

5. Conclusions

To underpin simulation models by empirical data means that one has to step into
methodological discussion, in particular into the question how deduction, induction
and abduction are related to each other. Most economists are educated in the tradition
of Positivism. As a consequence heterodox as well as mainstream economists pretend
- at least in their papers - that there are theoretical concepts they can deduce a priori
and then test them by confronting them with data. Despite the way economists
organize their papers it is correct to say that they also do not really deduce all abstract
concepts a priori in a first step but that they use empirical insights, mostly emerging
from a few observations interpreted by common sense in order to come to a theory

20
and then test this theory. What we argue in this paper is that these steps should be
made clear. Models should be based in a well-described way on empirical data.
Additional assumptions that are not based on empirical knowledge should be avoided
if possible or made at least explicit.

In order to calibrate simulation models we developed a methodology based on Critical
Realism. In order to do so we suggested using different sets of empirical data in two
ways by building on two strands of literature, which have already started to deal with
this problem. Firstly, in the literature on microsimulations empirical data is
extensively used for setting up the simulation model (see, e.g., Citro & Hanushek
1991). Secondly, in statistics Bayesian inference has been proposed to use empirical
data in order to detect the adequate models within a set of models (see Zellner 1971).
Both these strands of literature are mainly concerned with predicting future
developments. We, instead, aimed to infer statements about causal relations and
characteristics of a set of systems or dynamics, such as, e.g., the development of an
industry, that have a general validity for this set of systems or dynamics. In other
words, we aimed for general statement about the features of systems and dynamics
instead of probabilistic predictions about their future and for statement about a set of
systems or dynamics instead of an analysis of one single system. To this end, we
combine the procedures of microsimulations and Bayesian inference.

We have argued that the result of such a twofold use of empirics can be used in an
abductive way to create knowledge about classes of systems, where the classes can be
chosen according to different considerations. This leads us beyond the common use of
simulation model. We are able to infer from empirical data characteristics of classes
of systems that have a general validity. The examined characteristics might include
causal relationships as well as predictions of future developments. Hence, we are also
able to add to the understanding of economic processes. However, it is important to
realize that according to Critical Realism these results hold only temporarily, because
either the underlying mechanism might change in time or because better underlying
causal relationships are identified at a later stage.

21
References

Bagley, G., Burpee, J.E. and Jetté, S. (2000): Social Policy Microsimulation: What
Makes a Difference?, in A. Gupta and V. Kapur (eds): Microsimulation in
Government Policy and Forecasting, Amsterdam: Elsevier.

Brenner, T. (2004): Local Industrial Cluster: Existence, Emergence and Evolution.
London: Routledge.

Brenner, T. and J.P. Murmann (2003): The Use of Simulations in Developing Robust
Knowledge about Causal Processes: Methodological Considerations and an
Application to Industrial Evolution. Papers on Economics & Evolution #0303.

Citro, C.F. and Hanushek, E.A. (eds, 1991): Improving Information for Social Policy
Decisions: The Uses of Microsimulation Modeling, Vol. I, Review and
Recommendations, Washington D.C.: National Academy Press.

Cowan, R. and D. Foray (2002): Evolutionary economics and the counterfactual
threat: on the nature and role of counterfactual history as an empirical tool in
economics, in: Journal of Evolutionary Economics, 12 (5), 539-562.

Downward, P., J. H. Finch and J. Ramsay (2002): Critical realism, empirical methods
and inference: a critical discussion, in: Cambridge Journal of Economics, 26, 481-
500.

Eliasson, G., and E. Taymaz (2000): Institutions, Entrepreneurship, Economic
Flexibility and Growth – Experiments on an Evolutionary Micro-to-marco Model, in:
U. Cantner, H. Hanusch and S. Klepper (eds.): Economic Evolution, Learning, and
Complexity, Heidelberg: Springer-Verlag, 265-286

Ghosh, J.K. and R.V. Ramamoorthi (2003): Bayesian Nonparametrics, New York:
Springer.

22
Grupp, H. (1998): Foundations of the Economics of Innovation: Theory,
Measurement and Practice, Cheltenham: Edward Elgar Publ.

Malerba, F., R. Nelson, L. Orsenigo, and S. Winter (1999): `History-friendly' Models
of Industry Evolution: The Computer Industry, in: Industrial and Corporate Change,
8, 3-40.

Malerba, F., and L. Orsenigo (2002): Innovation and Market Structure in the
dynamics of the pharmaceutical industry and biotechnology: toward a History-
Friendly Model, in: Industrial and Corporate Change, 11, 667-703.

Kaldor, N. (1968): Capital Accumulation and Economic Growth, in: Lutz, F.A. and
D.C. Hague (eds.): "The Theory of Capital. Proceedings of a Conference Held by the
International Economic Association (1958). London: Macmillan Press, 177-222.

Kydland, F.E. and E. C. Prescott (1996): The Computational Experiment: An
Econometric Tool, in: Journal of Economic Perspectives, 10 (1), 69-85.

Lawson, T. (1997): Economics and Reality, London, New York: Routledge.

Machlup, F. (1978): Methodology of Economics and other Social Sciences, New
York, San Francisco, London: Academic Press.

Peirce, C. S. (1867) in Hartshorne, C. and P. Weiss (eds): Collected papers of Charles
Sanders Peirce, 1931-35, Vols 1-6, Cambridge (MA), US: Harvard University Press.

Richards, G. R. (2002): A Simulation Model with Endogenous Technical Advance:
Information Technology and Increasing Returns from Research, in: Technological
Forecasting & Social Change, 69, 29-51.

Schwerin, J. (2001): Wachstumsdynamik in Transformationsökonomien.
Strukturähnlichkeiten seit der Industriellen Revolution und ihre Bedeutung für
Theorie und Politik, Köln, Weimar, Wien: Böhlau Verlag.

23
Schwerin, J. and C. Werker (2003): Learning Innovation Policy Based on Historical
Experience, in: Structural Change and Economic Dynamics, December, 14, 385-404.

Zellner, A. (1971): An Introduction to Bayesian Inference in Econometrics, New
York: John Wiley.

W O R K I N G P A P E R S



Ecis working papers 2003 / 2004:



03.01 A. Nuvolari
Open source software development: some historical perspectives

03.02 M. van Dijk
Industry Evolution in Developing Countries: the Indonesian Pulp and Paper Industry

03.03 A.S. Lim
Inter-firm Alliances during Pre-standardization in ICT

03.04 M.C.J. Caniëls & H.A. Romijn
What drives innovativeness in industrial clusters?Transcending the debate

03.05 J. Ulijn, G. Duysters, R. Schaetzlein & S. Remer
Culture and its perception in strategic alliances, does it affect the performance? An exploratory study
into Dutch-German ventures

03.06 G. Silverberg & B. Verspagen
Brewing the future: stylized facts about innovation and their confrontation with a percolation model

03.07 M.C. Caniëls, H.A. Romijn & M. de Ruijter-De Wildt
Can Business Development Services practitioners learn from theories on innovation and services
marketing?

03.08 J.E. van Aken
On the design of design processes in architecture and engineering: technological rules and the principle
of minimal specification

03.09 J.P. Vos
Observing Suppliers observing Early Supplier Involvement: An Empirical Research based upon the
Social Systems Theory of Niklas Luhmann

03.10 J.P. Vos
Making Sense of Strategy: A Social Systems Perspective

03.11 J.A. Keizer & J.P. Vos
Diagnosing risks in new product development


03.12 J.M. Ulijn , A. Fayolle & A. Groen
European educational diversity in technology entrepreneurship: A dialogue about a culture or a
knowledge management class?

03.13 J.M. Ulijn, S.A. Robertson, M. O’Duill
Teaching business plan negotiation: How to foster entrepreneurship with engineering students

03.14 J.E. van Aken
The Field-tested and Grounded Technological Rule as Product of Mode 2 Management Research

03.15 K. Frenken & A. Nuvolari
The Early Development of the Steam Engine: An Evolutionary Interpretation using Complexity Theory

03.16 W. Vanhaverbeke, H. Berends, R. Kirschbaum & W. de Brabander
Knowledge management challenges in corporate venturing and technological capability building
through radical innovations

03.17 W. Vanhaverbeke & R. Kirschbaum
Building new competencies for new business creation based on breakthrough technological innovations

03.18 K.H. Heimeriks & G.M. Duysters
Alliance capability as mediator between experience and alliance performance: an empirical investigation
into the alliance capability development process

03.19 G.M. Duysters & K.H. Heimeriks
Developing Alliance Capabilities in a New Era

03.20 G.M. Duysters, K.H. Heimeriks, J. Jurriëns
Three Levels of Alliance Management

03.21 B. Verspagen & C. Werker
The invisible college of the economics of innovation and technological change

03.22 W. Vanhaverbeke, B. Beerkens, and G. Duysters
Explorative and exploitative learning strategies in technology-based alliance networks

03.23 S.J. van Dijk, G.M. Duysters & A.J.M. Beulens
Transparency dilemmas, information technology and alliances in agriculture and food industry

03.24 S.J. van Dijk & M.P.C.D. Weggeman
Knowledge sharing in technology alliances

03.25 C. Castaldi & A. Nuvolari
Technological Revolutions and Economic Growth:The “Age of Steam” Reconsidered

03.26 A. Nuvolari, B. Verspagen and N. von Tunzelmann
The Diffusion of the Steam Engine in Eighteenth-Century Britain

03.27 L. Wang & A.S. Szirmai
Technological Inputs and Productivity Growth in China’s High-Tech Industries

04.01 B. Nooteboom & V.A. Gilsing
Density and strength of ties in innovation networks: a competence and governance view

04.02 A. Nuvolari
Collective invention during the British Industrial Revolution: the case of the Cornish pumping engine


04.03 C. Meister & B. Verspagen
European Productivity Gaps: Is R&D the solution?

04.04 J.J. Berends, J.D. van der Bij, K. Debackere, M.C.D.P. Weggeman
Knowledge sharing mechanisms in industrial research

04.05 J.J. Berends, K. Debackere, R. Garud, M.C.D.P. Weggeman
Knowledge integration by thinking along

04.06 M.H.C. Ho
Differences between European Regional Innovation Systems in terms of technological and economic
caracteristics

04.07 F.E.A. van Echtelt, J.Y.F. Wynstra, A.J. van Weele van,., Duysters, G.M
Critical processes for managing supplier involvement in new product development: an in-depth multiple-
case study

04.08 H.A. Akkermans, I.S. Lammers, M.C.D.P. Weggeman
All ye need to know? Aesthetics from a design perspective

04.09 V. Gilsing & B. Nooteboom
Co-evolution in innovation systems: the case of pharmaceutical biotechnology

04.10 J.E. van Aken
Co-evolution in innovation systems: the case of pharmaceutical biotechnology

04.11 J.E. van Aken
Valid knowledge for the professional design of large and complex design processes

04.12 J.E. van Aken
Organising and managing the fuzzy front end of new product development

04.13 C. Werker & T. Brenner
Empirical calibration of simulation models