Establishing construct validity of a virtual-reality training
simulator for hysteroscopy via a multimetric scoring system
Michael Bajka Æ Stefan Tuchschmid Æ
Daniel Fink Æ Ga´bor Sze´kely Æ Matthias Harders
Received: 5 February 2009 / Accepted: 25 April 2009

Springer Science+Business Media, LLC 2009
Abstract
Background The aims of this study are to determine
construct validity for the HystSim virtual-reality (VR)
training simulator for hysteroscopy via a new multimetric
scoring system (MMSS) and to explore learning curves for
both novices and experienced surgeons.
Methods Fifteen relevant metrics had been identified for
diagnostic hysteroscopy by means of hierarchical task
decomposition. They were grouped into four modules
(visualization, ergonomics, safety, and fluid handling) and
individually weighted, building the MMSS for this study.
In a first step, 24 novice medical students and 12 experi-
enced gynecologists went through a self-paced teaching
tutorial, in which all participants received clearly stated
goals and instructions on how to carry out hysteroscopic
procedures properly for this study. All subjects performed
five repeated trials on two different exercises on HystSim
(exploration and diagnosis exercises). After each trial the
results were presented to the participants in the form of an
automated objective feedback report (AOFR). Construct
validity for the MMSS and learning curves were investi-
gated by comparing the performance between novices and
experienced surgeons and in between the repeated trials. To
study the effect of repeated practice, 23 of the novices
returned 2 weeks later for a second training session.
Results Comparing novices with the experienced group,
the ergonomics and fluid handling modules resulted in
construct validity, while the visualization module did not,
and for the safety module the experienced group even
scored significantly lower than novices in both exercises.
The overall score showed only construct validity when the
safety module was excluded. Concerning learning curves,
all subjects improved significantly during the training on
HystSim, with clear indication that the second training
session was beneficial for novice surgeons.
Conclusions Construct validity for HystSim has been
established for different modules of VR metrics on a new
MMSS developed for diagnostic hysteroscopy. Careful
refinement and further testing of metrics and scores is
required before using them as assessment tools for opera-
tive skills.
Keywords Virtual reality
Training
Simulation

Hysteroscopy
Evaluation
Construct validity

Learning curves
In the last decade, high-fidelity virtual-reality (VR) simu-
lators have emerged as valuable alternatives for practical
surgical skills training [1–6], excluding any risk to cause
harm to an individual [7]. Past efforts to incorporate sim-
ulation into surgical curricula for laparoscopy provide a
valuable roadmap on how a simulator for hysteroscopy
could be evaluated, validated, and finally integrated into
the training curriculum for gynecology [8].
As a proposed first step in the validation cascade [9],
face validity has been established with high ratings for both
realism and training capacity for HystSim [10], a new
surgery simulator for diagnostic and operative hysteros-
copy [11]. The presented results demonstrate that potential
trainees and trainers accept HystSim as a realistic and
useful tool for the training of hysteroscopic interventions.
M. Bajka (&)
D. Fink
Division of Gynaecology, Department of OB/GYN,
University Hospital Zurich, Zurich 8091, Switzerland
e-mail: michael.bajka@hin.ch
S. Tuchschmid
G. Sze´kely
M. Harders
ETH Zurich, Zurich 8092, Switzerland
123
Surg Endosc
DOI 10.1007/s00464-009-0582-4

As a second step of validation, construct validity is
usually investigated. Typically, it is established by com-
paring the performance for groups of surgeons with dif-
ferent degrees of experience [12–19]. The hypothesis is
tested that performance scores derived for a certain task on
the simulator are significantly higher for experts than for
novices.
While it is useful to know whether the different
parameters show construct validity, the final goal is to
judge and predict performance and ultimately the outcome
of an intervention. The first VR simulators in surgery (e.g.,
the MIST VR [20]) presented abstract tasks with geometric
bodies in a synthetic environment, using single criterions
such as time to complete a trial or counting of errors, for
both validation and assessment. However, it is doubtful
whether these common metrics are sufficient to assess
surgical performance comprehensively [21–25]. Recent
high-fidelity simulators present very realistic simulated
surgical scenes, implementing more and more combina-
tions of metrics and scoring systems which express com-
mon clinical skills, e.g., ‘‘economy of movements’’ [17] or
‘‘precision’’ [25]. Mackay found that the process of
assessing technical abilities is more robust if candidates are
tested on multiple parameters using a variety of measures
[26]. Van Dongen concluded that the implementation of a
scoring system enabled them to assess further aspects of
performance [25].
The selected metrics and the superimposed scoring and
grading system have to fulfill the following properties: (1)
clinical relevance—the metrics have to be as outcome
specific as possible, with clear reference to the underlying
goal of the procedure; (2) balance—the scoring system
should balance well between the sometimes conflicting
goals for the metrics, e.g., it should not be possible to
compensate low quality of performance with a short
intervention time; and (3) simplicity—the feedback should
be simple enough to be explained in a few seconds while
still providing useful and purposeful guidance for the
trainee.
Based on the characteristics above, it becomes clear that
each surgical procedure requires a customized scoring
system. While some of the metrics apply to most surgeries
and are employed by other simulators as well (e.g., inter-
vention time and instrument path lengths), others are
unique to hysteroscopy (e.g., time of insufficient expansion
of the uterine cavity and visualization of the tubal orifices).
Overviews of metrics used by different vendors of lapa-
roscopy simulators can be found in the literature [17, 27].
However, only a few of them have been validated rigor-
ously as assessment tools.
Therefore, analogously to Cao [28], we have performed
a hierarchical task decomposition of diagnostic hysteros-
copy [29], defining 4 tasks, 15 subtasks, 33 steps, and 46
substeps. This process resulted in the identification of 15
metrics for VR skills assessment which will be used here to
develop and validate a multimetric scoring system
(MMSS) for diagnostic hysteroscopy.
Thus, the main goal of this study was to explore con-
struct validity of HystSim, i.e., to what extent hysteroscopy
simulation in HystSim identifies the quality, ability, and
trait it was designed to measure [30]. Since the prime
motivation for using simulation is to accelerate the learning
of surgical skills, we were also interested in the learning
curves of trainees to find out more about the training effect
while using HystSim.
Materials and methods
Subjects
The group of novices consisted of 24 medical students with
no prior experience in hysteroscopy. They were recruited
by an email campaign to medical students in the fifth and
sixth years at the University of Zurich, Switzerland. In
addition, 12 gynecologists known as experienced hystero-
scopic surgeons with many years of practical experience
replied to an email invitation to participate in this study.
Apparatus
As in the previous face validity study [10], the HystSim
consisted of an adapted hysteroscope (10-mm resecto-
scope), a virtual patient robot, and the simulation software.
The simulation software ran on standard personal computer
(PC) hardware (dual 3.0-GHz Pentium processor, 2 GB
RAM, NVIDIA 8800 graphics card). The adapted resec-
toscope tracked all actions and movements of the trainee
and was used as input to adapt the simulation accordingly.
For this study, only the HystSim diagnostic hysteroscopy
software module (version 0.12) was employed. There was
no haptic feedback to guide the user in this version. Fig-
ure 1 shows screenshots of the running simulation and the
hardware setup used in this study.
Multimetric scoring system
Table 1 presents the metrics used in this study, together
with a short description of each parameter. Two surgical
experts, each having performed more than 500 hystero-
scopic interventions, were responsible for defining,
weighting, integration, and configuration of the metrics
into the scoring and grading system.
For diagnostic hysteroscopy, we grouped the parameters
into four modules, i.e., visualization, ergonomics, fluid
handling, and safety. The fluid handling module was only
Surg Endosc
123