t-e
Kent
Kriging
Nugget-effect
Metamodel
ly u
ut v
ty, f
get-
tion
es a
mation. The nugget-effect and proposed modified nugget-effect stabilize the estimated parameters and
y used
cteristi
ion pro
xperim
s the c
as the kriging model, is one of the more promising metamodels
as it is more flexible than regression models and not as compli-
cated and time consuming as artificial intelligence (AI) techniques;
see Li, Ng, Xie, and Goh (2010) for a comparative study. The kriging
model was originally developed in the field of geo-statistics; see
Matheron (1963). It was first introduced into Design and Analysis
of Computer Experiment (DACE) by Sacks, Welch, Mitchell, and
in a mining process. According to Cressie (1993, p. 127), the nug-
get-effect in geo-statistics is caused by two factors: micro-scale
variation and measurement error. In this article, we assume that
the system studied can be modeled as an L2-continuous random
process (see Cressie, 1993, p. 112), and hence the nugget-effect
studied here is purely caused by the random measurement error
(or random noise).
The nugget-effect in kriging assumes second-order stationarity
and is typically used to model white noise effect. Most kriging pub-
lications assume that the variance of the random error is homoge-
neous and the kriging model with nugget-effect is sufficient to
q This manuscript was processed by Area Editor Paul Savory.
⇑ Corresponding author. Tel.: +65 6516 3095; fax: +65 6777 1434.
Computers & Industrial Engineering xxx (2011) xxx–xxx
Contents lists availab
us
.eE-mail address: isensh@nus.edu.sg (S.H. Ng).lation model increases, the computing cost of running experiments
on the simulation model becomes much higher. Metamodels have
been applied as simplified approximations to the complex simula-
tion model; see Kleijnen (1987, 1998). Replacing the simulation
model with a metamodel in expensive experiments can increase
the efficiency and lower the computing costs. A review of
metamodel applications in engineering can be found in Simpson,
Peplinski, Koch, and Allen (2001). Among the different types of
metamodels available, the spatial correlation model, also known
ing predictions with the same values as the observations. For
example, in Gupta, Yu, Xu, and Reinikainen (2006), the kriging
metamodel is adopted for its interpolating characteristic. For sto-
chastic simulations where the responses at the same location vary
(for example in a simulation of a queueing system), the interpola-
tion characteristic of kriging models becomes less desirable. In or-
der to model the random fluctuations in stochastic situations, the
nugget-effect is introduced. The term ‘‘nugget’’ is borrowed from
geo-statistics, referring to the unexpected nugget of gold found1. Introduction
Computer simulation is commonl
aid in studying the system’s chara
especially useful in system optimizat
can be greatly reduced by running e
models instead of the real systems. A0360-8352/$ - see front matter  2011 Elsevier Ltd. A
doi:10.1016/j.cie.2011.05.008
Please cite this article in press as: Yin, J., et al. K
Engineering (2011), doi:10.1016/j.cie.2011.05.00decrease the erratic behavior of the predictor by penalizing the likelihood function affected by stochastic
noise. Several numerical examples suggest that the kriging model with modified nugget-effect outper-
forms the kriging model with nugget-effect and the classic kriging model in heteroscedastic cases.
 2011 Elsevier Ltd. All rights reserved.
in industry as a tool to
cs and behaviors. It is
blems, where the costs
ents on the simulation
omplexity of the simu-
Wynn (1989) and Sacks, Schiller, and Welch (1989). Recently, there
is an increasing interest in adopting kriging metamodels in indus-
trial engineering problems and applications (e.g. Ankenman,
Nelson, & Staum, 2010; Huang, Allen, Notz, & Zeng, 2006; Sakata,
Ashida, & Zako, 2007; Wang, Li, Li, & Zhong, 2008).
The kriging model is very suitable for deterministic simulation
problems. It is attractive for its interpolating characteristic, provid-Simulation
Heterogeneous variance
effects of stochastic noise on the parameter estimation for the classic kriging model that assumes deter-
ministic outputs and note that the stochastic noise increases the variability of the classic parameter esti-Kriging metamodel with modified nugge
variance caseq
J. Yin, S.H. Ng ⇑, K.M. Ng
Department of Industrial and System Engineering, National University of Singapore, 10
a r t i c l e i n f o
Article history:
Received 18 October 2008
Received in revised form 15 April 2011
Accepted 19 May 2011
Available online xxxx
Keywords:
a b s t r a c t
Metamodels are common
where the simulation outp
which assume homogenei
model with modified nug
model improves the estima
dent non-constant varianc
Computers & Ind
journal homepage: wwwll rights reserved.
riging metamodel with modifie
8ffect: The heteroscedastic
Ridge Crescent, Singapore 119260, Singapore
sed to approximate and analyze simulation models. However, in cases
ariances are non-zero and not constant, many of the current metamodels
ail to provide satisfactory estimation. In this paper, we present a kriging
effect adapted for simulations with heterogeneous variances. The new
s of the sensitivity parameters by explicitly accounting for location depen-
nd smoothes the kriging predictor’s output accordingly. We look into the
le at ScienceDirect
trial Engineering
lsev ier .com/ locate/caied nugget-effect: The heteroscedastic variance case. Computers & Industrial

solve the problem. However, there are many real world situations
where the homoscedastic assumption does not hold. These include
queueing systems and networks which can be found in many
industrial engineering problems. When applying the homoscedas-
tic kriging model in a heteroscedastic case, the fit can be poor,
especially when the sample size is small. We illustrate the noisy
applications with the simple function displayed in Fig. 1.
The test function consists of a second-order signal function and
a noisy function with step variance.
y ¼ SðxÞ þ e ¼ x2 þ e ð1Þ
where e indicates the random noise component, with variance
r2e ¼ 0:083 when x2[5, 2), and r2e ¼ 8:3 when x 2 [2, 5]. In Fig. 1,
the solid line indicates the signal function y = x2, and the dots are
the noisy observations of the signal function y = x2 + e.
In the traditional application of kriging in stochastic simula-
tions, replications are taken at each observation point and the aver-
ages of the replicates at each point are used as the inputs to the
model. Kleijnen (2008, p. 92) recommends at least nP 2 replica-
tions to be taken equally at each observation point when no prior
knowledge on the variance forms is available, otherwise, the sim-
ulation exercise may be meaningless due to the variability in the
data. In this test function example, we assume that a budget for
only 76 runs is available. Based on this, we spread 19 points from
5 to 5, taking four replicates at each point. The averages of the
four replicates at each of the 19 points are used as the inputs of
Gramacy and Lee, 2009; Gupta et al., 2006), limiting the number
of observation points and replications that can be taken.
Considering the kriging model with nugget-effect which has a
homogenous variance assumption, we pool the sample variances
at the 19 observation points to estimate the nugget-effect. The pre-
dictor output adopting this model is plotted as the dashed line in
Fig. 2.
As seen in Fig. 2, the nugget-effect predictor’s output is smoother
than the OK predictor. However, in the region x 2 [2, 5] where the
variance is higher, the fit is poor compared with the fit in the region
x2[5, 2). This indicates that the nugget-effect model can still be
inadequate as the heterogeneous variance can have an impact on
local predictions. Moreover, due to the homogeneous noise
assumptions of this model, there is no clear method to estimate
the nugget-effect under these heterogeneous conditions.
This same phenomenon occurs in the simulation of the M/M/1
queue, one of the most basic queueing models. Van Beers and
Kleijnen (2003) proposed a detrending approach to model out
the trend in the data using least squares methods and then apply
the deterministic ordinary kriging model to the detrended data.
Two alternative methods were later proposed by Kleijnen and
Van Beers (2005) to improve the application of kriging in stochastic
problems: the replication method and the studentization method.
The replication method proposes that the heteroscedastic problem
can be converted into a homoscedastic problem by taking appro-
priate replications at all the observation locations. This method re-
n y
isy
nal
2 J. Yin et al. / Computers & Industrial Engineering xxx (2011) xxx–xxxthe model. The solid line in Fig. 2 plots the fit of the traditional
deterministic ordinary kriging (OK) model.
With limited replications and input points, the ordinary kriging
model’s predictor output is poor with obvious fluctuations away
from the true function when x < 4 and x > 1. Because the tradi-
tional ordinary kriging model is designed under deterministic
assumptions, random noise can cause an ill fit and result in disap-
pointing predictions. We note that the predictor output will im-
prove as more replications and observation points are taken.
However, in many practical applications of simulation, the com-
puter model can be complicated and time consuming to run (see
-4 -3 -2 -1
ou
tp
ut
y
test functio
No
Sig
-5
-5
0
5
10
15
20
25
30inp
Fig. 1. Test function with s
Please cite this article in press as: Yin, J., et al. Kriging metamodel with modifie
Engineering (2011), doi:10.1016/j.cie.2011.05.008quires a sequential design with sufficient computing resources to
run all the replications. For example, in Fig. 1, the number of rep-
licates needed in the region with higher variance should be 100
times larger than the number of replicates in the region with lower
variance (because the variance is 100 times bigger in the former re-
gion) in order to convert the heteroscedastic case into a homosce-
dastic case. In the study of the M/M/1 queue system for the case
where the computing budget is limited, both the OK and nugget-
effect model with the application of this replication method can
still be inadequate. The studentization method is developed on
the basis of the detrended kriging approach. The main idea is to
0 1 2 3 4 5
=x2 with noise
observation
function onlyut x
tep variance function.
d nugget-effect: The heteroscedastic variance case. Computers & Industrial