Int. J. Human-Computer Studies
io
a,
sity of Y
ork,
d fo
ine
tera
me
ev
cal
d d
py
change when important characteristics present in the data vary.
r 2009 Elsevier Ltd. All rights reserved.
generating high-quality audio output are becoming widely
Interactive sonification can be considered as a particular
case of auditory display where the user is dynamically
Interaction in sonification can be implemented in two
be involved in producing the data.
ARTICLE IN PRESSIn both cases the aim of the interaction is to allow the
user to learn more, and more efficiently, about the
information being displayed.
1071-5819/$ - see front matter r 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.ijhcs.2009.05.006
Corresponding author. Tel.: +441904 433604; fax: +44 1904 432066.
E-mail addresses: sp148@york.ac.uk (S. Pauletto),
adh2@ohm.york.ac.uk (A. Hunt).available on everyday multimedia computing systems. The
discipline of sonification concerns the exploration of
methods for mapping the gathered data into a suitable
sonic form that can be readily comprehended by human
listeners.
1.2. Interactive sonification
main ways.
 Interactive sonification of recorded data: previously
stored data can be explored interactively while the
sound is being generated.
 Interactive sonification of data gathered in real time: the
sound can be produced as the data are gathered, i.e. in
real time. In this case, changes in the data are instantly
transformed into changes in sound and the user mightKeywords: Sonification; Audio feedback; Interactive navigation
1. Introduction
1.1. Sonification
Sonification is defined as ‘‘the use of non-speech audio
to convey information’’ (Kramer, 1994). It is considered to
be a timely topic, since just at the point in history where
data is being produced in abundance, the tools for
involved in the generation of the sound. Hermann and
Hunt define it as
‘‘ythe use of sound within a tightly closed human-
computer interface where the auditory signal provides
information about data under analysis, or about the
interaction itself, which is useful for refining the
activity.’’ (Hermann and Hunt, 2005, p. 20)Sandra Pauletto
aDepartment of Theatre, Film and Television, The Univer
bDepartment of Electronics, The University of Y
Received 16 September 2008; received in revise
Available onl
Abstract
In this paper we present two experiments on implementing in
(interactive navigation) and the second on data gathered in real ti
Complex synthesised data are explored in the first experiment to
distinguished using different interaction methods, while real medi
The addition of interaction to the exploration of sonified recorde
satisfaction), and the real-time sonification of complex physiotheraInteractive sonificatrm 26 February 2009; accepted 19 May 2009
30 May 2009
ction in sonification displays: the first focuses on recorded data
(auditory feedback).
aluate how well the known characteristics present in the data are
data (from physiotherapy) are used for the second.
ata improves the system usability (efficiency, effectiveness and user
data can produce sounds with timbral characteristics that audiblyHeAndy Huntb
ork, Heslington, York YO10 5DQ, United Kingdom
slington, York YO10 5DD, United Kingdom,67 (2009) 923–933
n of complex data
www.elsevier.com/locate/ijhcs

ARTICLE IN PRESS
n-C1.3. Review of research into interactive sonification
Interest in interaction has been shown by the auditory
display community since the early 1990s. The concept of a
Sound Probe (for pointing at a region of data of interest)
was presented in Gro¨hn (1992) and then used again in
Barrass and Zehner (2000) for the exploration of well logs.
Fernstro¨m and McNamara (1998) describe the importance
of direct manipulation (characterised by the continuous
representation of objects of interest and by immediate
feedback; Shneiderman and Plaisant, 2005) and apply it to
an application for browsing musical tunes.
Saue (2000) presented a general model for the sonifica-
tion of large spatial data sets in which ‘‘[y] the interpreter
is walking along paths in areas of the data set, listening to
locally and globally defined sound objects’’. This ‘‘virtual
walking’’ could be done using a mouse or similar input
device. The mouse has been the first choice of interface
device for many researchers working in interactive sonifi-
cation as it is the most common computer interface device
in use to this day. Winberg and Hellstro¨m (2001) used the
mouse as a virtual microphone. Hermann (2002) used it to
interact with data spaces in his early examples of Model-
Based Sonification (MBS)—a form of auditory display,
which is akin to configuring the data under investigation as
an instrument, which is ‘‘played’’ by the user in an
intrinsically interactive way.
In the last few years, with the higher processing power of
computers, more research on alternative interfaces can be
found. For example, Beamish et al. (2003) presented a
system that uses a haptic turntable for controlling the
playback of digital audio. They argue that the system,
initially intended for DJs, could also be used for the
exploration of data as sound. Hermann and colleagues
since 2001 have been exploring the use of novel interfaces
to interact with MBS systems. In Hermann et al. (2001) a
custom-built ‘‘hand box’’ interface is described where hand
posture is analysed and reconstructed as a multi-joint hand
model and used for exploration of sound and space. In
Hermann et al. (2003) the gesture desk interface is
introduced, which tracks the free movements of the hands
and uses them to interact with data spaces. Their later
audio-haptic ball is made of plasticine, and equipped with
various sensors (acceleration and force-sensitive sensors)
that send data to the computer when shaken, scratched,
squeezed, rotated and hit. These interactions are then used
to excite the MBS system, giving the user the impression
that they are shaking a box to discover what is inside.
Milczynski et al. (2006) presented a malleable interaction
surface for continuous and localised exploration of data
using the fingers, and Bovermann et al. (2006) showed how
moving an object (for example a stick) one could scan a
data set which is virtually positioned around the stick.
A good source of information on the various aspects of
interactive sonification can be found in the special issue of
S. Pauletto, A. Hunt / Int. J. Huma924IEEE Multimedia, which was dedicated, in 2005, to
interactive sonification (Hermann and Hunt, 2005).1.4. Interactive sonification of medical data
The medical community has already used sound feed-
back for a number of applications. The stethoscope is still a
fundamental medical tool, widely used for a range of tasks.
Clinicians use such interactive sonic feedback as an
everyday diagnostic technique, allowing them to detect
complex time-based events (such as heartbeat and air-
flow), and hidden structural defects; such is the power of
sound as an analytical aid. Sonification has the potential of
being able to portray many more simultaneous data
parameters than visual displays (Scaletti and Craig,
1990), whilst freeing the eyes and hands for visual–spatial
tasks (such as surgery, or communicating with colleagues
and patients). This requires the sound to be rendered in real
time, and reacting instantly to changes in the data.
Recent examples of studies on auditory feedback in the
medical field include Jovanov et al. (1998), who presented
the use of audio as feedback for precise manual positioning
of a surgical instrument and referred to it as tactile audio,
Effenberg (2005) on the auditory feedback of movement,
Hinterberger and Baier (2005) on auditory biofeedback of
electroencephalography (EEG) data that allows self-
regulation of the brain activity, Fox and Carlile (2005),
who presented the SoniMime system with auditory feed-
back to assist the user in learning a gesture with minimal
error and Ghez et al. (2000), who presented real-time
auditory feedback of joints and muscles for patients who
are unable to maintain ‘‘internal models’’ of their limbs and
therefore monitor movements. In all these examples, the
presence of audio was found to help self-regulatory
abilities, making it reasonable to hypothesise that auditory
feedback of physiotherapy data could produce interesting
results.
1.5. Experiments to investigate interactive sonification
The focus of this paper is to investigate the two main
scenarios for interactive sonification: recorded data and
data gathered in real time. In both cases we are
investigating if the addition of the real-time/interactive
element helps the analysis of the information contained in
the data. In both cases we are concentrating on large,
complex data sets, containing more than one channel of
data and with thousands of data points per channel. We
are investigating whether overall characteristics and
structures present in the data are clearly distinguishable
in the sound when (in the first case) the data are explored
with a high degree of interaction and (in the second case)
when the data are sonified in real time, at the original
sampling rate, producing a sound with a complex timbre.
The data used in the first experiment were synthesised by
the researcher so that the structures contained in the data
would be known. In the second experiment the data are
from the field of physiotherapy, a medical field where
omputer Studies 67 (2009) 923–933auditory biofeedback is of great interest for rehabilitation
or diagnostic tools.