ii
Roadmap  2007/4/26  16:00  page 1  #1
i
i
i
i
i
i

ii
Roadmap  2007/4/26  16:00  page 2  #2
i
i
i
i
i
i
This book was designed by Rebecka Bíró and Santos Miguel Tricio.
The text is set in the MS Garamond Family, while titles and headers
use MS Trebuchet.
It was typeset by Nicola Bernardini and Damien Cirotteau using the
L
A
T
E
X3 typesetting system on distributed GNU/Linux platforms.
Page geometries were checked using the Scribus Desktop Publishing
application v.1.3.3.4 (http://www.scribus.net).
Copyleft 2007 The S2S
2
Consortium
This work comes under the terms of the
Creative Commons BY 2.5 license
http://creativecommons.org/licenses/by/2.5/
The most recent version of this document may be downloaded from
http://www.soundandmusiccomputing.org/roadmap

ii
Roadmap  2007/4/26  16:00  page 3  #3
i
i
i
i
i
i
A Roadmap for
Sound and Music
Computing
Version 1.0
The S2S
2
Consortium
The most recent version of this document may be downloaded from
http://www.soundandmusiccomputing.org/roadmap

ii
Roadmap  2007/4/26  16:00  page 4  #4
i
i
i
i
i
i
The S2S
2
Consortium
Media Innovation Unit, Firenze Tecnologia, Firenze, Italy
Music Acoustics Group of the Kungliga Tekniska Högskolan in Stockholm,
Sweden
Music Technology Group of the Universitat Pompeu Fabra in Barcelona, Spain
CSC - Dept. of Information Engineering, University of Padova, Italy
Austrian Research Institute for Artificial Intelligence of the Austrian Society for
Cybernetic Studies in Vienna, Austria
Département d'Etudes Cognitives of the Ecole Normale Supérieure in Paris,
France
Laboratoire d'Etude de l'Apprentissage et du Développement of the Université de
Bourgogne in Dijon, France
Institute for Psychoacoustics and Electronic Music of the Universiteit Gent in
Ghent, Belgium
Laboratory of Acoustics and Audio Signal Processing of the Helsinki University of
Technology in Espoo, Finland
Vision, Image Processing & Sound Laboratory of the University of Verona, Italy
Laboratorio di Informatica Musicale of the University of Genova, Italy

ii
Roadmap  2007/4/26  16:00  page 5  #5
i
i
i
i
i
i
Coordinator Nicola Bernardini
Editors Xavier Serra
Marc Leman
Gerhard Widmer
Main Contributors Giovanni De Poli
Roberto Bresin
Vesa Välimäki
Davide Rocchesso
Alain De Cheveigné
Daniel Pressnitzer
Antonio Camurri
Emmanuel Bigand
Additional Contributors Federico Avanzini
Damien Cirotteau
Cumhur Erkut
Werner Göbl
Fabien Gouyon
Philippe Lalitte
Henri Penttinen
Pietro Polotti
Rainer Typke
Gualtiero Volpe

ii
Roadmap  2007/4/1  16:00  page 6  #6
i
i
i
i
i
i

ii
Roadmap  2007/4/1  16:00  page 7  #7
i
i
i
i
i
i
Executive Summary
Music is an important aspect of all human cultures. Music is meant to
give new experiences, to give sense and meaning to life, to console and to
promote social coherence and personal identity in and across very diverse
social and ethnic groups. Rooted in the biology of every human being,
music is a core occupation of our technological society. By 2020, music will
have become a commodity as ubiquitous as water or electricity. Its content
and the activities surrounding it will promote new business ventures,
which in turn will bolster the music and cultural/creative industries.
Sound and Music Computing (SMC) will provide the core technologies
for this ongoing revolution in electronic music culture. Its major research
contribution to advances in the field will be to bridge the semantic gap,
the hiatus that currently separates sound from sense. This contribution
will stimulate fruitful interaction between culture, science and industry.
Sound and Music Computing (SMC) research can be traced back to the 1950's,
when a handful of composers, together with engineers and scientists, began ex-
ploring the use of the new digital technologies for the creation of new music and
multimedia content. This new field of research had a profound impact on the
development of culture and technology in our postindustrial society. Since then
SMC has not only made great advances in a variety of areas that range from dig-
ital signal processing to cognitive musicology, but has also contributed to many
successful technological applications, ranging from the synthesis engines of digital
musical instruments to the audio CD, MP3 and polyphonic ringtones.
Today, SMC research is Europe's most advanced multidisciplinary approach to
music and multimedia. By combining scientific, technological and artistic metho-
dologies it aims at understanding, modelling and producing sound and music us-
ing computational approaches. SMC focuses on how cultural content can be in-
tegrated with ICT and other innovative technologies.
A recent study of the economic impact of the cultural and creative sector in
Europe
1
revealed that its annual turnover (e654 billion in 2003) is larger than
that of the motor industry or even ICT Manufacturers. This sector, of which the
music industry is a major part, contributed 2.6% of EU GDP in 2003, slightly
more than the contribution of the chemicals, rubber and plastic products indus-
try combined. The sector's growth in 19992003 was 12.3% higher than that of
1
KEA Study http://www.kernnet.com/kea/Ecoculture/ecoculturepage.htm

ii
Roadmap  2007/4/1  16:00  page 8  #8
i
i
i
i
i
i
the general economy and in 2004, about 5.8 million people worked in it, equiv-
alent to 3.1% of the total employed population in EU25. In view of the Euro-
pean Council's Lisbon agreement of March 2000, that the EU by 2010 should
become `the most competitive and dynamic knowledgebased economy in the
world, capable of sustainable economic growth with more and better jobs and
greater social cohesion, reinforced coordination of activities and European Com-
mission policies impacting on the cultural and creative sector should be given a
high priority.
The main objective of this SMC roadmapping project is to identify, characterise
and propose strategies for tackling the key research challenges that this grow-
ing and diversified domain will be facing in the next ten to fifteen years. This
Roadmap should help overcome the present fragmentation of Europe's efforts
in the area of SMC by establishing a common agenda and ensuring consolida-
tion, integration and exploitation of research results from European initiatives
and projects. Hence, this project is clearly positioned at the strategic science and
research roadmapping level: the expected result is not a roadmap focused on
a particular product or technology (the most common type of roadmap), but
rather the definition of a strategic programme for SMC.
This Roadmap is targeted at the whole SMC community. It should be specially
useful to researchers in both academia and industry, giving them a wide perspec-
tive on their own research work. It should also be relevant to educators and pol-
icy makers, informing them of the key issues that should be emphasised in train-
ing and taken account of when making funding decisions.
This SMC Roadmap includes three scenarios illustrating how our everyday life
could change by 2020 if the challenges were met, a definition of the field, a de-
scription of the research, educational, industrial, and social/cultural contexts, a
state of the art overview identifying the current key research issues and a strate-
gic pathway proposal with five challenges: to design better sound objects and environ-
ments, to understand, model, and improve human interaction with sound and music, to train mul-
tidisciplinary researchers in a multicultural society, to improve knowledge transfer, and to address
social concerns.
This document is the result of two years of work by the S2S
2
Consortium. In-
novative working methodologies and procedures were used. These included the
concentration of the efforts of a board of senior researchers directly involved
in the S2S
2
Consortium, the creation of an advisory board to gather consensus
and qualified advice from the SMC community at large, and finally the running
of several iterative cycles of internal and public review.

ii
Roadmap  2007/4/6  16:00  page i  #9
i
i
i
i
i
i
Preface
Over the past 50 years, music and technology have forged such a strong con-
nection that all aspects of the economic chain, from production to distribution
and consumption, have become digital. The real sound and sense for music is
situated at the endpoints of this chain, when musicians play, or when listeners
search for and enjoy music as active consumers. All the rest is electronically en-
coded, virtual, and difficult to access. But it works! Music has become a major
ecommerce commodity, paving the way for new business models and innovative
applications in mobile technology and many other fields.
Music is the language of our emotions, of social bonding, of personal develop-
ment and intellectual enrichment. But access to technology will be the major
means by which this language can be fully explored. Indeed, the next revolu-
tion is about the connection between sound and sense; that is, the connection
between encoded physical energy in technology and the human subjective ex-
perience. New technology is needed to close this huge gap. This new technol-
ogy will impact on how people normally have access to music at all levels of the
digital economic chain. It will revolutionise how people deal with music, trans-
forming recording and broadband technology for sound and music into a vast,
worldwide, allpenetrating musical instrument that all humans can easily access.
This revolution will be taken up by the dynamic forces that drive music, which
in turn will lead to new developments and opportunities in the cultural and cre-
ative industries. New innovative products will foster social interaction among
people and it will also create new opportunities by offering new tools for ex-
pression and communication, from contentbased music information retrieval,
to interactive music systems for artistic production, to soundaware daily envi-
ronments.
This revolution has a bright future, but it needs strong support and coordinated
action. At this moment, Europe is playing a leading role in research that aims at
bridging this gap between sound and sense. The intention of this roadmap is to
set out the basic strategic directions needed to coordinate research activity in this
domain.
This Roadmap is one of the results of the S2S
2
Coordination Action
2
. Apart
from it being the greatest of honours for me to coordinate this Consortium 
representing the top European researchers in this field  I must say that it has
proved to be a most pleasurable experience. I wish to thank all my colleagues
both for their herculean efforts and also for their unfailing kindness, patience
2
The S2S
2
(Sound to Sense, Sense to Sound) project was submitted to the European Commission
during the 6th Framework Programme in the Future and Emergent Technologies (FET)Open
call series and its funding has run from June 2004 to May 2007. Other notable results of the S2S
2
Action have been:
• the S2S2: A State of the Art in Sound and Music Computing book (in press)
• the Sound and Music Computing Summer School
The S2S
2
website is the official source of information concerning the S2S
2
project.
i

ii
Roadmap  2007/4/1  16:00  page v  #13
i
i
i
i
i
i
Sound and Music Computing
6. Conclusions 95
A. Notes for SMC Curricula 99
1.1 A Survey of SMC Education . . . . . . . . . . . . . . . . . . . . . . 101
1.2 Results of the Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
1.2.1 Distribution of content areas . . . . . . . . . . . . . . . . . . 105
1.2.2 Content co-occurrence analysis . . . . . . . . . . . . . . . . . 107
1.3 Towards Unified Curricula in SMC . . . . . . . . . . . . . . . . . . . 110
B. Resources Available for SMC Research 113
2.1 Data Collection Process . . . . . . . . . . . . . . . . . . . . . . . . . 114
2.1.1 Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
2.1.2 Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . 114
2.1.3 Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
2.2 SMC Research centres . . . . . . . . . . . . . . . . . . . . . . . . . . 118
2.3 SMC Conferences, Journals and Societies . . . . . . . . . . . . . . . . 124
2.3.1 Conferences . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
2.3.2 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
2.3.3 Academic Societies . . . . . . . . . . . . . . . . . . . . . . . . 131
2.4 Analysis of Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
2.4.1 Size of the Field (Budgets, Personnel) . . . . . . . . . . . . . . 132
2.4.2 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
2.4.3 Startups and Patents . . . . . . . . . . . . . . . . . . . . . . . 137
2.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
C. SMC European Projects 139
v

ii
Roadmap  2007/3/19  16:00  page vi  #14
i
i
i
i
i
i

ii
Roadmap  2007/3/20  16:00  page 1  #15
i
i
i
i
i
i
CHAPTER 1
Scenarios from the Future: The
Benefits of Sound and Music
Computing Research
The three scenarios presented in this chapter represent visions of life after a few
attainable (though not necessarily easy) scientific/technological targets have been
hit through the removal of roadblocks, the filling in of gaps and the meeting of
certain challenges as outlined in Chap.5.
The scenarios describe general environments and activities concerning our ev-
eryday life soundscapes, the professional perspective of musicians and general
music appreciation. As such, there is no onetoone correspondence between
a particular scenario and any of the particular key issues identified in Chap.4.
Rather, they provide hints of how the world could be if and when Sound and
Music Computing research achieves the multidisciplinarity and transversality pro-
posed in this Roadmap.
1

ii
Roadmap  2007/3/20  16:00  page 2  #16
i
i
i
i
i
i
Chapter 1. Scenarios from the Future
1.1 CONTROLLABLE SOUND ENVIRONMENTS
Sensors, actuators, microprocessors, and wireless connection facilities are increas-
ingly being embedded into everyday objects. These can be augmented with sonic
features that make the environment more enjoyable, social life more interesting,
and personal life more relaxed and healthy.
Grandma and me
Hi, I'm a teenager  15 years old  and I like to wake up late, at least on Sun-
days. But today I decided to spend a few hours with my grandmother, so I set
my alarm clock bed for 9 a.m. First, the bed tries to wake me up gently, by vi-
brating and making purring noises. Even though I am not sleeping anymore, I
like to wait until the bed gets more nervous, when it realises I am still lying on
it, and starts making harsh rhythmic movements and squeaking. I love it!
I get up and go into the living room. It's a mess after last night's party. You know,
mum and dad are on vacation, so ... Chairs are everywhere; chips and peanuts
all over the floor. I play some of my favourite hiphop music while I put things
back in place and prepare for the day. Different MCs and DJs are embodied in
different pieces of furniture. I know it sounds retro, but it is so cool to move
DJ Grandmaster Flash while I am moving that old armchair around. While I
move through the house, the music seamlessly follows me through the objects
I pass. When I leave home, I put on my headset and keep riding the beat. The
headset doesn't cut me off from the environment,though. When I catch sight of
a strange bird singing, I look at it and put my hand to my ear. This gesture acti-
vates acoustic zooming, and I can appreciate the bird song in isolation. But then
I am distracted by the sight of a friend of mine chatting to a girl. Instinctively, I
steer my acoustic zoom towards them, but I realize she is wearing one of those
new active jackets that can create an acoustic shield around you. I wonder what
they are talking about.
I reach my grandma's house. She became almost deaf about five years ago, but
she is really brave and decided to get an implant of powerful bionic ears. In re-
cent years, she has become more and more worried about the bad things that
could happen to her. That's why my dad bought her a new door. As we leave
the house and she closes the door, she proudly explains that the complex sound
of the lock tells her that the lights are switched off, the gas is turned off, the
fish are fed, and the window in the living room has been left wide open. I'm
sure she left it open on purpose, but we go back in and close it anyway.
We are going to the Fred Astaire club today. Grandma is wearing brand new
Mike shoes. She feels much more confident about herself in these shoes, because
they give her bionic ears some sonic feedback about equilibrium so she's not
afraid of falling while she moves around. Before we go into the club she wants
2

ii
Roadmap  2007/4/1  16:00  page 3  #17
i
i
i
i
i
i
Sound and Music Computing
to check on her health, which is promptly sonified through her bionic ears. Ev-
erything sounds fine, so we go in just as the show's starting. There are a dozen
overeighties there, wearing Mike shoes, tap dancing, and clearly having a good
time. Their subtle, gentle movements trigger a massive and diverse set of rhy-
thms. Who knows  it may even be cooler than hip hop!
On the way back home, grandma tells me how different the town was when she
was a child. There was even a working water mill. Fortunately, we can both en-
joy its sound. Both my cheap headset and her bionic ear can induce selective si-
lence and let the lost sounds of the town emerge from history. That makes her
remember even more. It's fun being with grandma.
3

ii
Roadmap  2007/4/1  16:00  page 4  #18
i
i
i
i
i
i
Chapter 1. Scenarios from the Future
1.2 MUSIC INSTRUMENTS FOR ALL
In 2020, many sound devices will have a general purpose computer in them and
will include quite a number of realtime interaction capabilities, sensors and wi-
reless communication. Basically, any soundproducing device will be able to be-
have like a personalised musical instrument. Music making will become perva-
sive and many new forms of communication with sound and music will become
available. Music content will be inherently multimodal and music making avail-
able to everyone: music from all to all.
How I became a professional musician
A year ago I bought the new wearable mobile device from SMC Inc. With it I
was able to listen to my favourite songs and interact with them in ways not pos-
sible with the previous generation of devices. Now I was able to change many
aspects of the songs by gestural and vocal control. Some of my friends were re-
ally good at it, and I started to improve my skills by practising on my home mul-
timedia system. This system includes Jeeves, a virtual musical assistant, which
observes and analyses my body movement, my singing and my musical abilities
in general. Jeeves teaches me how to express myself in the style of famous mu-
sicians, from The Beatles to cellist YoYo Ma. After a period of training, I was
ready to play and jam with other users over the Internet and get advice from
more expert users or their virtual musical assistants. After a couple of months,
I started to get a good reputation in the community. One day, a group of users
asked me to join them in person at a discotheque. In that discotheque we were
able to use our mobiles to plug into a music role in the overall show. People
took all kinds of roles; some were projected as visual characters on the surround-
ing space and walls, some were projected into moving lighting, others, like me,
were controlling the expressivity of the music being played. Since I was a be-
ginner, I took a simple role as the controller of the timbral aspects of the drum.
Another person took control of the drum sticks. We had to dance to coordinate
the rhythm in this shared drum set with the other visual and musical roles. This
experience felt much more physical and exciting than being at home with my
multimedia system. In the discotheque I had the feeling of being part of a com-
munity and of real teamwork. The various haptic devices in my clothes height-
ened my aural and visual perception and interaction with friends.
I met these new friends many times; I practised a lot in discotheques and at home.
I developed my own style and developed good skills in controlling virtual instru-
ments, with Jeeves evolving with me and my community. I could control expres-
sion and lead my friends in improvisations and jam sessions. One day Jeeves
asked me: Could I please change my name to Madonna? I feel that my back-
ground knowledge has changed. I realised that was true and changed her name.
Today I am a professional musician: an MJ (Music Jockey). Madonna and I pre-
4

ii
Roadmap  2007/3/20  16:00  page 5  #19
i
i
i
i
i
i
Sound and Music Computing
pare the shows on the authoring system at home. I design the structure, frame-
work, roles, and musical material to be used. In my shows, I sometimes impro-
vise with acoustic instrument players. I collect data on my sound device by mon-
itoring their movements and performance choices. I can also monitor and anal-
yse all the events in the show and the behaviour of everyone involved in it, in-
cluding the audience. My virtual musical assistant Madonna wants to change her
name again. Now she would like to be called Karajan.
5

ii
Roadmap  2007/3/20  16:00  page 6  #20
i
i
i
i
i
i
Chapter 1. Scenarios from the Future
1.3 PERSONALISED MUSIC DEVICES
Current portable mp3 players  despite their simplicity  have already radi-
cally changed human music-listening behaviour. Now, the first webbased music
information systems which provide contextual information about music simply
by connecting already existing services (such as Wikipedia, CDDB, lastFM,etc.),
without the utilisation of any musical expertise, are beginning to emerge. Based
on current trends in SMC research, we predict that such systems are likely to
further develop in the direction of multimodal, interactive, open and adaptive
systems that support both beginners and experts from different cultures in ac-
cessing music and musicrelated information.
My new music friend
I take my expert music companion with me anywhere, anytime, because I love
music. The companion doesn't just play music. It gives me a lot of other infor-
mation about the music  from `practical' things such as transcriptions of in-
struments and harmonies, to animated visualisations of the structure of the mu-
sic, contextual information such as style, historical and cultural relations, and the
relationship of the piece to other, related pieces and styles.
My device is easy to use. I can talk to it, or I can shake it to show it the kinds
of rhythms I like. It is aware of the music being played on radio stations and
available in music databases worldwide, and it finds new music that I like in a
particular situation. I can point it at music being played by a street band, and it
will tell me what it is. It understands my intentions and learns my musical pref-
erences. Sometimes it will surprise me, teaching me something new about music
and my taste. And by the way, having had nanosized loudspeakers (painlessly)
implanted in my ears, I listen to my music without bothering with bulky head-
phones and earplugs.
My music companion also helps me out in social contexts. When I am desper-
ately looking for a date, my companion alerts me there's a dance party around
the corner for people with a similar interest in Brazilian music. When I get to it,
my companion contacts the DJ system and sends it some of my favourite pieces
(rare Brazilian stuff). The girl in the corner just goes Wow.
My music companion is no longer an isolating device that runs playlists; it's a
friend that enhances my musical abilities, reflects my personality and helps me
to socialise.
6

ii
Roadmap  2007/4/1  16:00  page 11  #25
i
i
i
i
i
i
Sound and Music Computing
2.1 DISCIPLINES INVOLVED
The disciplines involved in SMC cover both human and natural sciences. Its core
academic subjects relate to musicology, physics (acoustics), engineering (includ-
ing computer science, signal processing and electronics), psychology (including
psychoacoustics, experimental psychology and neurosciences) and music compo-
sition.
Musicology
All research that deals with musical meaning formation, musical content descrip-
tion, and associated mediation technologies in particular sociocultural contexts. It
comprises the study of how musical content can be described and how the sub-
jective and sociocultural background of users plays a role in the production, dis-
tribution and consumption of music.
Physics
Acoustics is the science concerned with the production, control, transmission,
reception and effects of sound as a physical phenomenon. The branch of acous-
tics of special interest to the SMC community is music acoustics. It includes the
acoustics of musical signals (such as in expressive music performance), musical
instruments and singing voices.
Engineering
This includes all the research in computer science and engineering, signal pro-
cessing and electronics that deals with music information representation, process-
ing and communication. It comprises multimedia information systems, artificial
intelligence, audio signal processing, robotics, sensors and interface technology.
Psychology
This includes all research into musicrelated behaviour and brain processes, in-
cluding the roles of perception, cognition, emotion and motor activities. Psychol-
ogy is here understood to cover the whole domain from psychoacoustics to ex-
perimental psychology to neurosciences.
Music Composition
This concerns all research that has a focus on musical content creation. It in-
cludes creating music as a score, as an interactive artistic event, as a sound instal-
lation, as a soundtrack, and as any form of organised sound event which com-
municates information.
11

ii
Roadmap  2007/4/1  16:00  page 12  #26
i
i
i
i
i
i
Chapter 2. Definition of The Field
2.2 AREAS OF APPLICATION
Most SMC research is of the applied kind and thus possible applications have an
important role in the definition of the field. Current areas of application include
digital music instruments, music production, music information retrieval, digital
music libraries, interactive multimedia systems, auditory interfaces and augmented
action and perception (e.g. bionic ears, digital prosthesis and multimodal exten-
sions of the human body).
Digital music instruments
This application focuses on musical sound generation and processing devices. It
encompasses simulation of traditional instruments, transformation of sound in
recording studios or at live performances and musical interfaces for augmented
or collaborative instruments.
Music production
This application domain focuses on technologies and tools for music composi-
tion. Applications range from music modelling and generation to tools for music
postproduction and audio editing.
Music information retrieval
This application domain focuses on retrieval technologies for music (both au-
dio and symbolic data). Applications range from music audioidentification and
broadcast monitoring to higherlevel semantic descriptions and all associated
tools for search and retrieval.
Digital music libraries
This application places particular emphasis on preservation, conservation and
archiving and the integration of musical audio content and metadata descrip-
tions, with a focus on flexible access. Applications range from large distributed
libraries to mobile access platforms.
Interactive multimedia systems
These are for use in artistic, entertainment and infotainment applications. They
aim to facilitate musicrelated humanmachine interaction involving various mo-
dalities of action and perception (e.g. auditory, visual, olfactory, tactile, haptic,
and all kinds of body movements) which can be captured through the use of au-
dio/visual, kinematic and bioparametric (skin conduction, temperature) devices.
Auditory interfaces
These include all applications where nonverbal sound is employed in the com-
munication channel between the user and the computing device. Auditory dis-
plays are used in applications and objects that require monitoring of some type
of information. Sonification is used as a method for data display in a wide range
12

ii
Roadmap  2007/4/1  16:00  page 13  #27
i
i
i
i
i
i
Sound and Music Computing
of application domains where auditory inspection, analysis and summarisation
can be more efficient than traditional visual display.
Augmented action and perception
This refers to tools that increase the normal action and perception capabilities
of humans. The system adds virtual information to a user's sensory perception
by merging real images, sounds, and haptic sensation with virtual ones. This has
the effect of augmenting the user's sense of presence, and of making possible a
symbiosis between her view of the world and the computer interface. Possible
applications are in the medical domain, manufacturing and repair, entertainment,
annotation and visualisation, and robot teleoperation.
13

ii
Roadmap  2007/4/1  16:00  page 14  #28
i
i
i
i
i
i
Chapter 2. Definition of The Field
2.3 ACADEMIC SUPPORT
Academic support for the SMC field has not yet reached a mature state. There
are a number of related academic societies, conferences and journals, but only a
few of them have a clear focus on SMC issues, reflecting the fact that SMC is
not a well- established academic field. Nevertheless, the situation has improved
in the last decade, especially in Europe. Below we list the major international
academic societies, journals and conferences related to SMC. A more complete
list can be found in Appendix B.
2.3.1 Academic societies
International Computer Music Association (ICMA); International Musicological
Society (IMS); Acoustical Society of America (ASA); European Acoustics Asso-
ciation (EEA); Institute of Electrical and Electronics Engineers (IEEE); Associ-
ation for Computing Machinery (ACM); Audio Engineering Society (AES); Eu-
ropean Society for the Cognitive Sciences of Music (ESCOM); Society for Music
Perception and Cognition (SMPC) and Electroacoustic Music Studies Network
(EMS)
2.3.2 Journals
Computer Music Journal; Journal of New Music Research; EURASIP Journal on
Audio; Speech; and Music Processing; IEEE Signal Processing Magazine; IEEE
Signal Processing Letters; IEEE Transactions on Audio; Speech and Language
Processing; IEEE Multimedia Magazine; Journal of the Acoustical Society of Ame-
rica; Acta Acustica; united with Acustica; Journal of the Audio Engineering Soci-
ety; Musicae Scientiae; Music Perception; Psychology of Music; Leonardo Music
Journal; Computing in Musicology; Perspectives of New Music and Organised
Sound
2.3.3 Conferences
International Computer Music Conference (ICMC); International Conference on
Music Information Retrieval (ISMIR); International Conference on Digital Au-
dio Effects (DAFx); International Conference on New Interfaces for Musical
Expression (NIME); International Conference on Music Perception and Cogni-
tion (ICMPC); Sound and Music Computing International Conference (SMC);
ACM Multimedia; Meeting of the Acoustical Society of America; AES Conven-
tions and International AES Conferences; International Conference on Music
and Artificial Intelligence (ICMAI); IEEE International Conference on Acous-
tics; Speech and Signal Processing (ICASSP); IEEE Workshop on Applications
14

ii
Roadmap  2007/4/3  16:00  page 21  #35
i
i
i
i
i
i
Sound and Music Computing
complexity of what is observed. Some recent studies in neurophysiology include
the use of awake preparations (often coupled with behavioural studies), multi-
ple unit recordings, simultaneous invasive and noninvasive brain imaging tech-
niques (to calibrate one with respect to the other), selective brain cooling, optical
imaging and the coupling of one of these with genetic engineering or biochemi-
cal manipulations to probe specific stages in processing. Research in brain imag-
ing includes the use of higher magnetic fields for structural and functional MRI
(magnetic resonance imaging), increased numbers of channels in EEG (electroen-
cephalography) or MEG (magnetoencephalography), simultaneous recording of
fMRI and EEG, or EEG and MEG, and use of presurgical supradural or intra-
cortical recordings from patients to obtain `close up' snapshots of brain activity.
An important facilitating factor in these developments is progress in hardware
and software techniques for handling and interpreting the massive data sets pro-
duced by brain imaging. In short, there is presently a rapid development of dif-
ferent technologydriven methodologies that provide new insights into how the
brain is involved in the semantic gap problem.
Statement 4: New technologydriven methodologies are providing new insights into
the human brain.
A third major research effort in theoretical neurosciences is about a tight inter-
action between signal processing and machine learning techniques on the one
hand, and models of neural processing on the other. A common goal is to find
techniques that can harness the extreme complexity of relevant patterns in data
(for example databases of environmental, speech or musical sounds) or the struc-
tures and mechanisms observed within the brain. The computer here is used as
an aid to harness a degree of complexity of which our brains cannot otherwise
easily comprehend. One promising angle of enquiry is the use of datadriven de-
velopment of the processing mechanism (natural or artificial) under the drive of the
data patterns that it is to process, as an alternative or complement to more tradi-
tional engineering techniques.
On the practical side, one can speculate on possible future benefits from the neu-
rosciences. An example of such a hypothetical breakthrough might be the possi-
bility of `downloading' entire cognitive or perceptual processing mechanisms to
software. This could result from a combination of progress in recording tech-
niques, theoretical neurosciences and machine learning. Another hypothetical
breakthrough (heralded by the wellestablished cochlear implant and recent ex-
periments with animal models and impaired humans) could be the widespread
development of brainmachine interfaces (BMI). This could result from a com-
bination of progress in interface hardware (e.g. miniaturised electrode arrays),
signal processing (to factor out `noise') and machine learning (to translate be-
tween the different codes used by brain and machine). All this research is likely
to have a huge impact on the SMC field. Examples are hearing aids that allow
21

ii
Roadmap  2007/4/3  16:00  page 22  #36
i
i
i
i
i
i
Chapter 3. Contexts
their users to listen to music or an intracortical implant that would allow a quad-
riplegic to play the piano.
Statement 5: Cognitive and neurosciences offer a rapidly expanding window on the
human mind and brain, thereby providing new possibilities for solving the semantic
gap problem.
Trend 3: From subjective experience to cultural content in the
Humanities
Research in the humanities is focused on signification practices; that is, on how
human being make sense of their environment and give meaning to their lives.
The humanities view this signification practice from a subjective and experiential
point of view. Therefore, research of this kind includes anthropology, area stud-
ies, communications, cultural studies and media studies. The humanities not only
provide insights into these aspects but also train people in the skills necessary
for practitioners (e.g. in music playing, painting). Traditionally, research methodo-
logies in the humanities are based on analytic, descriptive, critical or even specu-
lative and imitation approaches, although recent approaches also involve quantita-
tive and empirical studies [14, 15, 16]. In the cultural and creative industries, the
humanities can provide the content needed to develop a significant partnership
between culture and technology.
Several research efforts in the humanities address this issue.
A first approach has adopted the belief that subjective factors in (related to gen-
der, education and social and cultural background) play a central role in how peo-
ple deal with technology. Knowledge of these factors needs to be incorporated
into music information retrieval technologies. Humanities research provides the
necessary analysis of the social and cultural context in which technological appli-
cations will function.
Statement 6: Subjective factors play a central role in how people deal with technol-
ogy.
A second research approach is concerned with what is sometimes called `medi-
alogy'; that is, an approach which combines technology and creativity to design
new processes and tools for art, design and entertainment. It involves insight
into the creative processes, thoughts and tools needed for mediaproductions
and other arts to exist. Clearly, medialogy is at the crossroads of the human sci-
ences, the creative arts and technology. As such, it is a central pillar of the cre-
ative industries.
22

ii
Roadmap  2007/4/3  16:00  page 25  #39
i
i
i
i
i
i
Sound and Music Computing
as an encompassing discipline that studies a phenomenon of central relevance to
humans in all its necessary breadth. In addition, the emergence of new multidis-
ciplinary fields of research and application is producing new points of contact
for SMC.
A prime example of such contact is the current rise of the socalled creative indus-
tries (CI). While the notion creative industries refers to a sector of the econ-
omy, its current upsurge (also in terms of public awareness) also leads to new
opportunities for creative multidisciplinary research at the intersection of art, de-
sign and technology. SMC can and will play an important role here.
The case of the creative industries also highlights once more  if that were needed
 the close ties between scientific research and the arts. Artistic visions coupled
with creative application ideas are likely to drive SMC research in more ways than
can currently be envisioned, resulting in entirely new environments, devices and
cultural services.
Statement 9: Multidisciplinary research is increasingly seen as a necessity and an
asset, and special programmes for fostering and funding it are being developed.
SMC can take advantage of this and should actively seek alliances with other
disciplines, including the arts.
References
[1] P. (Ed.) Hengeveld, J.-P. Best, J. van Beumer, B. Hooff, H. van den Poot, and R. de Westerveld.
Research trends in information and communication technology: Uncovering the agendas for the
information age. 2000.
[2] ITRS Consortium. The International Technology Roadmap for Semiconductors. http://www.
itrs.net/home.html, 2007.
[3] NEM Consortium. Networked and Electronic Media  European Technology Platform: Strate-
gic Research Agenda. http://www.nem-initiative.org/, 2007.
[4] Microsoft Research Cambridge. Towards 2020 Science. http://research.microsoft.com/
towards2020science/, 2006.
[5] Robert A. Wilson and Frank Keil, editors. The MIT Encyclopedia of the Cognitive Sciences (MITECS).
The MIT Press, Cambridge, Mass, 1999.
[6] National Health Institute. Interdisciplinary Research. http://nihroadmap.nih.gov/
interdisciplinary, 2006.
[7] Natural Sciences and Engineering Research Council. First Report of the Advisory Group on
Interdisciplinary Research. http://www.nserc.ca/pubs/agir/AGIR_e_report.pdf, 2002.
[8] National Science Foundation. Converging Technologies for Improving Human Performance: Nanotechnol-
ogy, Biotechnology, Information Technology and Cognitive Science. Kluwer Academic Publishers, 2003.
[9] European Commission. European Research Area. http://cordis.europa.eu/era/.
25

ii
Roadmap  2007/4/1  16:00  page 28  #42
i
i
i
i
i
i
Chapter 3. Contexts
potential of SMC technologies (see the IndustrialContext section). New curricula
in SMC have the opportunity to address these emerging labour markets.
A major recent change in higher education has been the increase in student mo-
bility. A considerable part of overall mobility is supported through the EC's E-
rasmusSocrates programme [7], established in 1987, which seeks to reinforce
the European dimension of higher education by encouraging transnational co-
operation between universities and boosting European mobility. The figures for
mobility reflect a steady improvement, but remain below what the Commission
considers necessary [3]. Moreover, the EU still attracts less talent than its com-
petitors. [2]. SMC research in Europe has a successful track record involving
excellence spread over several centres which have gained world leadership through
complementarity and coordination supported by EC funding. This excellence
has to be exported to the higher education domain, in order to attract students,
scholars and researchers from other world regions.
Statement 1: New trends of higher education in Europe give more possibilities for
designing curricula in SMC.
Trend 2: Discipline oriented undergraduate education
The tradition of undergraduate education is very much discipline oriented. A
student has to choose a curriculum aimed at developing a number of specific
competences in a particular discipline plus a few general academic and profes-
sional competences. However there are curricula in Europe that are more multi-
disciplinary or that allow a student a wider choice of itineraries, thus permitting
the design of `custom made' curricula. With respect to research, the involvement
of undergraduate students in such activities as a normal part of their curriculum
is still very exceptional.
Given that there are many academic disciplines integral to SMC research (for a
detailed discussion see the content areas listed in Appendix A), the education
given in all the undergraduate degrees supporting these disciplines are of inter-
est to any future SMC researcher. Thus a student wanting to become an SMC
researcher might choose an undergraduate degree related to musicology, physics,
computer science, electrical engineering, psychology, music composition, etc...
Within most of the undergraduate programmes that support these disciplines,
there are specific courses that might be of very great relevance to SMC. But in
most cases it really depends on the professor responsible for the course and the
special focus given to it.
Statement 2: Numerous paths, embedded in different wellestablished undergrad-
uate degrees, can be designed to approach an multidisciplinary field such as SMC.
28

ii
Roadmap  2007/4/1  16:00  page 29  #43
i
i
i
i
i
i
Sound and Music Computing
The music conservatories are a special case of higher education institutions in
the context of SMC. Traditionally, they have a strong professional orientation
and thus might not provide the necessary background for a musician wanting
to follow a research career. This situation has been slowly changing, due both
to the transformations taking place in the music profession and also, in Europe,
to the inclusion of the conservatories into EHEA [6]. Slowly, the conservatories
are converging with the university system. It is now becoming quite common for
a conservatory to offer a degree with a strong technological component. There
are, for example, conservatory degrees in sound recording, tonmeister, sonol-
ogy, music technology, electroacustic music, etc... Most of these degrees remain
professionally oriented but very much related to SMC. Conservatories are also
slowly incorporating the idea of research as one of their institutional aims and
are designing curricula which are closer to the university model.
Statement 3: New conservatory degrees are a model for professionally oriented
undergraduate curricula in SMC.
Apart from the traditional university degrees and the case of the music conserva-
tories, there are quite a number of multidisciplinary undergraduate programmes
related to SMC, especially in the US and Great Britain. In the AngloSaxon sys-
tem, it is much easier for universities to establish multidisciplinary programmes
or even to allow studentcentred curricula with individual academic pathways.
However, there is an ongoing discussion among academics and researchers about
the type of undergraduate education best suited to preparation for a research ca-
reer in an multidisciplinary field like SMC a strongly discipline oriented under-
graduate degree or an multidisciplinary programme.
The adoption of a common system of credits, such as the ECTS system, plus
the existence of funding programs like Erasmus to support mobility have had a
big impact on undergraduate education. They have led students to become fa-
miliar with other approaches to a given field and have given them the oppor-
tunity to take courses not offered in their 'home' university. The Erasmus pro-
gramme has also facilitated the creation of networks of universities with comple-
mentary undergraduate degrees in a given discipline, so that experiences among
faculty members can be shared and the curricula opportunities for students are
widened. Due to the variety of disciplines and methodological approaches in-
volved in the SMC field, it is not easy to find educational institutions with an ex-
tensive coverage of all of them. It is thus very useful for an undergraduate stu-
dent wanting to get a wider view of the field to take courses in different centres.
Statement 4: Undergraduate degrees with multidisciplinary contents encourage stu-
dent mobility.
29

ii
Roadmap  2007/4/1  16:00  page 32  #46
i
i
i
i
i
i
Chapter 3. Contexts
• Consistency with established knowledge in multiple separate disciplinary
antecedents: how the work stands vis à vis what researchers know and find
tenable in the disciplines involved.
• Balance in weaving together perspectives: the extent to which the work
hangs together as a generative and coherent whole.
• Effectiveness in advancing understanding: the extent to which the integra-
tion of disciplinary perspectives advances the goals that have been set and
the methods used.
Statement 7: The traditional model of a Master/apprentice relationship in PhD
studies is evolving in a much more complex education environment, especially for
multidisciplinary fields like SMC.
The need for more structured PhD studies in Europe and the relevance of such
studies to the Bologna Process have been highlighted repeatedly in recent years.
In particular, joint PhD programmes can be amongst the most attractive features
of the EHEA. But for the time being, interested students are still confronted
with a variety of national and institutional structures that are not easily compa-
rable.
Statement 8: Joint SMC PhD programmes at the EU level can be built by
exploiting excellence spread over several centres with complementary competencies.
Attention to employable skills and competencies in doctoral programmes is in-
creasing. There is a clearly growing trend towards the professionalisation of PhD
studies, involving the inclusion of coursework and training in transferable skills
aimed at facilitating the flow of doctoral students into the wider job market. Stu-
dents are becoming employed researchers within wellstructured research groups
and funded within wellfocused research projects. Within this context, PhD stu-
dents represent major academic and financial investments and contribute to much
of the original research in universities. The role of supervisors seems key to the
success or failure of multidisciplinary PhD projects [11]. There is clear evidence
that the disciplinary background, interest and motivation of the supervisor have
much influence on research outcomes, both in terms of its quality and also whe-
ther PhD studies are completed on time (or at all).
However the addedvalue of a PhD for employment outside the areas of research
in universities, research institutes and R&D functions in industry remains some-
what limited. Central and East European countries especially, as well as South
European countries, experience a continuing lack of interest on the part of em-
ployers outside the academy in hiring PhDs. The situation is almost reversed in
the US, where a significant and ever growing number of PhDs are attracted to
private sector employment in which remuneration is higher than in the academy [10].
32

ii
Roadmap  2007/4/1  16:00  page 33  #47
i
i
i
i
i
i
Sound and Music Computing
Statement 9: multidisciplinary PhD programmes avoid a focus which is too narrow
and provide a broad spectrum of knowledge that also qualifies their graduates for
careers outside the academy.
References
[1] Trends IV: European universities implementing Bologna, 2005. http://www.eua.be/eua/jsp/en/
upload/TrendsIV_FINAL.1117012084971.pdf.
[2] European Commission. Progress Towards the Lisbon Objectives in Education and Training,
2005. http://europa.eu.int/comm/education/policies/2010/doc/progressreport05.pdf.
[3] Education and Training 2010 - Diverse Systems, Shared Goals. http://europa.eu.int/comm/
education/policies/2010/et_2010_en.html.
[4] C. Tauch. Almost Half-time in the Bologna Process Where Do We Stand? European Journal of
Education, 39(3):275288, 2004.
[5] European Commission. The Bologna Process. Towards the European Higher Education Area.
http://ec.europa.eu/education/policies/educ/bologna/bologna_en.html.
[6] European Association of Conservatoires (AEC). The Bologna Declaration and Music. http:
//www.bologna-and-music.org/.
[7] European Commission. Socrates  Erasmus. The European Community programme in the field
of higher education. http://ec.europa.eu/education/programmes/socrates/erasmus/erasmus_
en.html.
[8] European Commission. Erasmus Mundus. http://ec.europa.eu/education/programmes/
mundus/index_en.html.
[9] N. Metzger and R. Zare. Science Policy: Interdisciplinary Research: From Belief to Reality. Sci-
ence, 283:642643, 1999.
[10] J. Sadlak. Doctoral Studies and Qualifications in Europe and the United States: Status and
Prospects. Technical report, UNESCO-CEPES, 2004.
[11] G. Fry, B. Tress, and G. Tress. PhD students and integrative research. In Proc. Frontis Workshop
From Landscape Research to Landscape Planning: Aspects of Integration, Education and Application Wa-
geningen, The Netherlands, June 2004.
33

ii
Roadmap  2007/4/1  16:00  page 35  #49
i
i
i
i
i
i
Sound and Music Computing
accounts for over 20% of OECD value added. The share of high and medium
high technology manufacturing fell to about 7.5% of total OECD value added in
2002, compared to about 8.5% in 2000.
Statement 1: Music related activities are part of the new knowledge economy and
they should take advantage of the continuing growth of this sector.
Trend 2: A Global economy
Economies have expanded beyond national borders. Production in particular
has been expanded by multinational corporations to many countries around the
world. The global economy includes the globalisation of production, markets, fi-
nance, communications and the labour force.
From the OECD report [1] we learn that this is not a new phenomenon per se,
but that it has become more pervasive and driven mainly by the use of informa-
tion and communication technologies (ICT). In the knowledge economy, infor-
mation circulates at the international level through trade in goods and services,
direct investment and technology flows and the movement of people.
According to the American National Science Board [4] the globalisation of R&D,
S&T, and S&E labour markets is continuing. Countries are seeking competitive
advantage by building indigenous S&T infrastructures, attracting foreign invest-
ments and importing foreign talent. The location of S&E employment is becom-
ing more internationally diverse and those who are employed in S&E have be-
come more internationally mobile.
Statement 2: Both the production and consumption of music related goods is now
globalised and international cooperation is more important than ever.
Trend 3: The development of the ICT sector
In the final decade of the twentieth century, the almost simultaneous arrival of
mobile phones and the Internet not only changed the face of communications
but also gave impetus to dramatic economic growth. We now speak of the In-
formation and Communication Technologies sector to refer to the agglomeration
of the communications sector, including telecommunications providers and the
information technology sector, which ranges from small software development
firms to multinational hardware and software producers.
35

ii
Roadmap  2007/4/1  16:00  page 36  #50
i
i
i
i
i
i
Chapter 3. Contexts
According to the i2010 report [2] ICT accounts for a quarter of EU GDP growth
and 40% of productivity growth. The digital convergence of the information so-
ciety and media services, networks and devices is finally becoming an everyday
reality: ICT will become smarter, smaller, safer, faster, always connected and eas-
ier to use, with content moving to three- dimensional multimedia formats.
Roberto Saracco [3] points out that any economic indicator ties together pro-
gress and communications infrastructure, and that the dissemination and pro-
gress of culture go hand in hand with the possibility of interacting and sharing
ideas, thus putting telecommunications at centre stage.
The American National Science Board [4] reports that the number of industrial
researchers has grown along with rapidly increasing industrial R&D expenditures.
Across OECD member nations, employment of researchers by industry has grown
at about twice the rate of total industrial employment. For the OECD as a whole,
the fulltime equivalent number of researchers more than doubled in the two
decades from 1981 to 2001, from just below 1 million to almost 2.3 million. Over
the same period, the number of researchers in the United States rose from 0.5
million to nearly 1.1 million.
According to the KEA report [10] the ICT sector is central to European growth
and competitiveness and has been identified as a pillar of the European Lisbon
Strategy. It accounts for 5.3% of EU GDP and 3.4% of total employment in
Europe. In the period 20022003 it was responsible for more than a quarter
both of productivity growth and of the total European R&D effort.
Damon Darlin [7] predicts that flatscreen televisions will get bigger and that
MP3 players and cell phones will get smaller. And almost everything will get cheaper.
But the biggest trend expected is that these machines will communicate with one
another.
According to the OECD report [8], digital music and other digital content are
also drivers of global technology markets, both to consumer electronics manufac-
turers and PC vendors. The increase in revenues from hardware in the PC and
consumer electronics branch resulting from the availability of online music, au-
thorised or not, is potentially greater than the revenues currently generated by
paid music streaming or downloads.
Statement 3: The growth of the ICT sector and the innovations coming out of it
will be the main driving forces for the music related industries.
36

ii
Roadmap  2007/4/1  16:00  page 37  #51
i
i
i
i
i
i
Sound and Music Computing
Trend 4: The interdependence of the cultural & creative sector
and ICT
The cultural and creative sector generates significant economic performance in
other noncultural sectors, thereby indirectly contributing to economic activity
and development, and in particular in the ICT sector.
Culture contributes directly to the economy by providing products for consump-
tion, namely the cultural goods and services embodied in books, films, music
sound recordings, concerts, etc. But the recent growth of the creative media,
according to the KEA report [10], is due to the growing diffusion and impor-
tance of the Internet. The impact of this development on media consumption
has been huge in recent years and will be the major factor for this sector in the
future. At the same time, creative content is a key driver of ICT uptake. The
consultancy firm PriceWaterhouseCoopers estimates that spending on ICT- re-
lated content will account for 12% of the total increase in global entertainment
and media spending until the year 2009.
The development of new technology depends to a large extent on the attractive-
ness of content and the new networks are no exception. The development of
mobile telephony and networks is based on the availability of attractive value
added services that will incorporate creative content.
However, the KEA report [10] predicts that the roll out of broadband and the
digitisation of production processes will require significant investment for the
creative industries to adapt, as well as changes in its management practices. Some
industries (notably music) have had to go through aggressive cost restructuring
programmes and are experiencing consolidation through mergers. Without a strong
music, film, video, TV and game industry in Europe, the ICT sector will be the
hostage of content providers established in Asia or North America.
Statement 4: Content is a major driver of ICT development.
Trend 5: New models of exploitation of content
The new ICT technologies have opened up new possibilities for the exploita-
tion of music. Traditionally there have been two distribution channels for media
content: physical distribution and analog broadcasting (radio, TV). Now we also
have: IP/Internet, Mobile communications (UMTS), Digital TV and Radio.
The OECD report [8] identify that network convergence and widespread diffu-
sion of highspeed broadband have shifted attention towards broadband con-
tent and applications that promise new business opportunities, growth and em-
ployment. Digital content and digital delivery of content and information are be-
37

ii
Roadmap  2007/4/1  16:00  page 38  #52
i
i
i
i
i
i
Chapter 3. Contexts
coming increasingly ubiquitous, driven by the increasing technological capabilities
and performance of delivery platforms, the rapid uptake of broadband technolo-
gies  with 2004 identified as a breakthrough year for broadband penetration in
OECD countries  innovative creation and use of content and improved per-
formance of hardware and software. Through a combination of new technolo-
gies, new business relationships and innovative service offers to consumers, the
market is developing rapidly in order to realise the potential of online music.
Roberto Saracco [3] predicts that in ten years' time nearly all communications
(over 90% of it) will be using fixed networks, while most people will be under
the impression they are using mobile networks. He observes that in the coming
years we are going to see a tremendous increase in communicating entities, be
they applications or objects. The amount of communication directly involving
humans will keep growing but at a slower pace, fuelled mostly by the dissemina-
tion of telecommunications in developing countries.
We read in the OECD report [9] that users are becoming increasingly active, that
we are entering a participatory culture not of consumers but of users. Users are
increasingly active and want to express themselves.
Statement 5: Interactive broadband networks are revolutionising the way music is
distributed and consumed.
Trend 6: New forms of Intellectual Property protection
Traditionally, there are only two extreme positions: absolute control of a creation
or complete release of the rights to it. Until recently, there was no easy way to
make explicit the rights that an author gives in relation to a creation. All the ini-
tiatives that explore alternatives to the traditional copyright are called copyleft.
(Creative Commons [11] is the first example of a system that offers flexible pro-
tection of intellectual rights).
David Kusek and Gerd Leonhard [5] claim that the issue of protecting intellec-
tual property goes far beyond music and audio technologies. Nevertheless, the
crisis started in the music industry. Already, music recording industry revenues
are down sharply, despite an overall increase in the distribution of music. The fi-
nancial crisis has caused music labels to become cautious and conservative, in-
vesting in proven artists, with less support available for new and experimental
musicians. Kusek and Leonhard, in their Manifesto for the Digital Music Rev-
olution, [5] note that the breakdown of copyright protection is even starting to
impact on musical instruments. Synthesisers, samplers, mixers and audio proces-
sors can all be emulated in software. For example they estimate that at least 90%
of the copies of "Reason," one of the emulation software leaders, are pirated.
38

ii
Roadmap  2007/4/6  16:00  page 40  #54
i
i
i
i
i
i
Chapter 3. Contexts
to catapult music to unprecedented heights. Today, the Internet and other dig-
ital networks, despite all the legal wrangling, have made music bigger than ever
before. Within ten to fifteen years, Kusek and Leonhard claim, the Music Like
Water business model will make the industry two or three times larger than it is
today. They imagine a world where music flows all around us, like water or elec-
tricity, and where access to music becomes a kind of "utility". Not for free per
se, but certainly for what feels like free.
Kurzweil [6] claims music technology is about to be radically transformed. Com-
munication bandwidths, the shrinking size of technology, our knowledge of the
human brain and human knowledge in general are all accelerating.
Music will remain the communication of human emotion and insight through
sound from musicians to their audience, but the concepts and process of music
will be transformed once again.
Statement 7: The possibilities of the ICT technologies are completely reshaping the
music business.
References
[1] OECD. OECD Science, Technology and Industry Scoreboard - Towards a knowledge-
based economy. Organisation for Economic Co-operation and Development. http://hermia.
sourceoecd.org/vl=849972/cl=36/nw=1/rpsv/scoreboard/, 2005.
[2] Information Society and Media DG. i2010 - A European Information Society for growth and
employment. http://europa.eu.int/information_society/eeurope/i2010/index_en.htm.
[3] R. Saracco. Is there a future for Telecommunications? http://www.singaren.net.sg/library/
presentations/21nov02_1.pdf, 2002.
[4] National Science Board. Science and Engineering Indicators 2006. Technical report, National
Science Foundation, 2006. http://www.nsf.gov/statistics/seind06/.
[5] D. Kusek and G. Leonhard. The Future of Music: Manifesto for the Digital Music Revolution. Omnibus
Press, 2005. http://www.futureofmusicbook.com/.
[6] Ray Kurzweil. The Future of Music in the Age of Spiritual Machines. Lecture to the AES 2003.
http://www.kurzweilai.net/meme/frame.html?main=/articles/art0597.html.
[7] D. Darlin. Data, music, video: Raising a curtain on future gadgetry. New York Times, January
2006. http://tinyurl.com/njg3f.
[8] OECD. OECD Report on Digital Music: Opportunities and Challenges. Technical report, 2005.
[9] OECD. The Future Digital Economy: Digital Content Creation, Distribution and Access. Tech-
nical report, 2006. http://www.oecd.org/dataoecd/54/35/36854745.pdf.
[10] KEA European Affairs. The Economy of Culture in Europe. Technical report. http://www.
keanet.eu/Ecoculture/ecoculturepage.htm.
[11] Creative Commons. http://creativecommons.org/.
40

ii
Roadmap  2007/4/6  16:00  page 44  #58
i
i
i
i
i
i
Chapter 3. Contexts
Trend 4: A Neoevolutionary research model
Given the broad context in which audio and music manifest themselves, SMC re-
search strategies are characterised by emergence rather than planning. This emer-
gence, moreover, is driven by creativity and innovation. Hence it is difficult to
predict what may be successful and what not. SMC's scientific paradigm is there-
fore close to a neoevolutionary model [5], in which elaborate systems of peer
review, assessment and evaluation leave room for strategies of variation to be
pursued by smaller laboratories in different alliances.
Statement 5: SMC research is strongly driven by innovation, albeit in a context
of emergence rather than planning.
In this model, risk analysis is needed to consider the possible implications of re-
search. After all, science and technology do not automatically lead to the best
possible world. In developing them, it is necessary to calculate the risks, to keep
an eye on the volatile and ambiguous dynamics. The coevolution of the socio
cultural context and the scientific/technological context implies that an analysis
of values and goals should become an integral part of the development of SMC
[7]. The best guarantee to cope with unpredictable outcomes, or uncertainties
initiated by innovation, is to allow society and culture to speak back to science
and technology, hence the importance of reflection, the development of a code
of ethics, the concern for democratic access and several other values that should
be taken into account.
Statement 6: Democratic access, reflection and a code of ethics should form an
integral part of SMC research.
Trend 5: Innovation through artistic creation
Creation and innovation form the motor of SMC research. Most interestingly,
they are strongly driven by the context of artistic application. In that respect, it
is of interest to mention that contentbased music technology has roots in the
particular cultural rationality of the 1950s [1], [4]. That rationality, heavily sup-
ported by European governments of the time, led to novel developments in elec-
tronic music, of which interactive multimedia is a recent outcome. In contrast,
audiorecording technology had already begun by the early 20th Century and
was driven by the logic of economic rationality and the free market [8].
The trend of allying contentbased music technology to economic rationality is
new. But it is reasonable to assume that artistic creation remains a major factor
44

ii
Roadmap  2007/4/6  16:00  page 51  #65
i
i
i
i
i
i
Sound and Music Computing
domain techniques as a step towards developing sound models that might be
used for retrieval, synthesis or recognition applications.
Also of importance has been the study of soundproducing physical objects.
The aim of such study is to understand the acoustic characteristics of musical in-
struments and other physical objects which produce sounds relevant to human
communication. Its main application has been the development of physical mod-
els of these objects for synthesis applications [4, 5, 10], so that the user can pro-
duce sound by interacting with the models in a physically meaningful way.
However, beyond the physical aspect, sound is a communication channel that
carries information. We are therefore interested in identifying and representing
this information. Signal processing techniques can only go so far in extracting
the meaningful content of a sound and in the past few years there has been an
exponential increase in research activity which aims to generate semantic descrip-
tions automatically from audio signals. Statistical Modelling, Machine Learning,
Music Theory and Web Mining technologies have been used to raise the seman-
tic level of sound descriptors. MPEG7 [11] has been created to establish a fra-
mework for effective management of multimedia materials, standardising the de-
scription of sources, perceptual aspects and other relevant descriptors of a sound
or any multimedia asset.
Most research approaches in sound description are essentially bottom up, start-
ing from the audio signal and trying to reach the highest possible semantic level.
There is a general consensus that this approach has clear limitations and does
not allow us to bridge what is known as the semantic gap. The current trend
is towards Multimodal Processing methods and topdown approaches based on
Ontologies, Reasoning Rules, and Cognition Models. For example, collaborative
tagging by users is being increasingly used to attach semantic information to pic-
tures.
Key Issues
Perceptual versus motorbased models
Perceptual systems are usually studied separately from motor systems, but there
are strong arguments in favour of merging the two, or at least for including the
dimension of action within the study of perception. It has been argued, for ex-
ample, that visual information is not accrued by sampling successive 2D patterns
from the retina, but rather from the interplay between sensory changes and eye,
body and environmental movements. The structure of threedimensional space
can be `learned' from the crosscorrelation between movement and sensory in-
formation. It has even been claimed that perceptual information is, to some de-
gree, stored internally in the form of `potential actions'. The technological coun-
terpart in robotics is the joint development of sensory and motor functions, or
the `calibration' of delicate control mechanisms based on sensory feedback [9].
51

ii
Roadmap  2007/4/6  16:00  page 53  #67
i
i
i
i
i
i
Sound and Music Computing
els [4, 5, 15], or at capturing the perceptual characteristics of the sound signal,
generally referred to as spectral or signal models [3]. The technology transfer ex-
pectations of the physical models have not been completely fulfilled. Their ex-
pressiveness and intuitive control  advantages originally attributed to this kind
of model  did not help commercial music products to succeed in the market
place. Meanwhile, synthesis techniques based on spectral modelling have met
with competitive success in voice synthesisers, both for speech and singing voi-
ces, but to a lesser extent in the synthesis of all other musical instruments. A
recent and promising trend is the combination of physical and spectral models
such as physically informed sonic modelling [2] and commuted synthesis [4, 5].
The new corpusbased concatenative methods for musical sound synthesis, also
known as mosaicing, have attracted much attention recently [14]. They make use
of a variety of sound snippets in a database to assemble a desired sound or phrase
according to a target specification given in sound descriptors or by an example
sound. With everlarger sound databases easily available, together with a perti-
nent description of their contents, these methods are increasingly used for com-
position, highlevel instrument synthesis, interactive exploration of a sound cor-
pus and other applications. In sound processing, there are a large number of ac-
tive research topics. Probably the most well established are audio compression
and sound spatialisation, both of which have clear industrial contexts and quite
well defined research agendas. Digital audio compression techniques allow the
efficient storage and transmission of audio data, offering various degrees of com-
plexity, compressed audio quality and compression itself. With the widespread
uptake surge of the mp3, audio compression technology has spread to main-
stream audio and is being incorporated into most sound devices. These recent
advances have resulted from the understanding of the human auditory system
and the implementation of efficient algorithms in advanced DSP processors. Im-
provements to the state of this art will not be easy but there is a trend towards
trying to make use of our new understanding of human cognition and of the
sound sources to be coded.
Sound spatialisation effects attempt to widen the stereo image produced by two
loudspeakers or stereo headphones, or to create the illusion of sound sources
placed anywhere in three dimensional space, including behind, above or below
the listener. Some techniques, such as ambisonics and wavefield synthesis, are
readily available and new models are being worked on that combine signaldriven,
bottomup processing with hypothesisdriven, topdown processing [1].
Digital sound processing also includes the techniques used for audio postpro-
duction and other creative uses in music and multimedia applications [7]. Time
and frequency domain techniques have been developed for transforming sounds
in different ways and in a number of other applications. But the current trend is
to move from signal processing to content processing; that is, to move towards
higher levels of representation for describing and processing audio material.
53

ii
Roadmap  2007/4/6  16:00  page 54  #68
i
i
i
i
i
i
Chapter 4. The State of the Art
There is a strong trend towards the use of all these signal processing techniques
in the general field of interactive sound design. Sound generation techniques have
been integrated in various multimedia and entertainment applications (sound ef-
fects and background music for gaming), sound product design (ring tones for
mobile phones) and interactive sound generation for virtual reality or other mul-
timodal systems. Old sound synthesis technologies have been brought back to
life and adapted to the needs of these new interactive situations. The importance
of control has been emphasised, and sourcecentred and perceptioncentred mod-
elling approaches have been expanded towards interactive sonification [6].
Key Issues
Interactioncentred sound modelling
The interactive aspects of music and sound generation should be given greater
weight in the design of future sound synthesis techniques. A challenge is how to
make controllability and interactivity central design principles in sound modelling.
It is widely believed that the main missing element in existing synthesis tech-
niques is adequate control modelling. Feature and expressive content extraction
from human gestures, from haptic (for example pressure, impacts or frictionlike
interactions on tangible interfaces) to movement (motion capture and analysis) to
voice (extraction of expressive content of the voice or breath of the performer),
should become the main paradigm for new research in sound generation. This
development also opens the field to multisensory and crossmodal interaction re-
search. Besides the analysis and coding of the expressivity of the human body
`playing', the consequent problem concerns how to exploit the extracted con-
tents in order to model sound. Effective sound generation needs to achieve a
perceptually robust link between gesture and sound. The mapping problem is in
this sense crucial both in musical instruments (see also Section 4.2.1) and in any
other device/artefact involving sound as one of its interactive elements.
Modular sound generation
Sound synthesis by physical modelling has, so far, mainly focused on accurate
reproduction of the behaviour of musical instruments. Some other efforts have
been devoted to everyday sounds or to the application of sophisticated numeri-
cal methods for solving wave propagation problems. A seductive dream has been
that of a toolkit for constructing sounding objects from elementary blocks such
as waveguides, resonators and nonlinear functions. The dream has faced a num-
ber of intrinsic limitations in blockbased descriptions of musical instruments. In
general, it is difficult to predict the sonic outcome of an untested connection of
blocks. However, by associating macroblocks to salient phenomena, it should
be possible to devise a constructivist approach to sound modelling. At the low-
est level, blocks should correspond to fundamental interactions (impact, friction,
air flow on edge, etc.). The sound quality of these blocks should be tunable, ba-
54

ii
Roadmap  2007/4/1  16:00  page 59  #73
i
i
i
i
i
i
Sound and Music Computing
Integration of control with sound generation
The separation between gesture controllers and output generators has some sig-
nificant negative consequences, the most obvious being the reduction of the `feel'
associated with producing a certain kind of sound. Another frequent criticism
is the inherent limitations of MIDI, the protocol that connects these two com-
ponents of the instrument chain. However, there is a more basic drawback con-
cerning the conceptual and practical separation of potentially new digital instru-
ments into two separated components. This is that it becomes hard  or even
impossible  to design highly sophisticated control interfaces without a pro-
found prior knowledge of how the sound or music generators will work. Gene-
ric or nonspecific music controllers tend to be either too simple, mimetic (imi-
tating traditional instruments) or too technologically biased. They can be inven-
tive and adventurous, but their coherence cannot be guaranteed if they cannot
anticipate what they are going to control [15].
Feedback systems
When musicians play instruments, they perform certain actions with the expec-
tation of achieving a certain result  a musical performance. As they play, they
monitor the behaviour of their instrument and, if the sound is not quite what
they expect, they will adjust their actions to change it. In other words, they have
effectively become part of a control loop, constantly monitoring the output from
their instrument and subtly adjusting bow pressure, breath pressure or whatever
control parameter is appropriate. The challenge is how to provide the performer
of a digital instrument with the appropriate feedback to control the input param-
eters better than that provided by mere auditory feedback. One proposed solu-
tion is to make use of the musician's existing sensitivity to the relationship be-
tween an instrument's `feel' and its sound with both haptic and auditory feed-
back [18]. Other solutions rely on visual and auditory feedback [15].
Designing effective interaction metaphors
Beyond the two previous issues, which concern the musical instrument paradigm,
the design of structured and dynamic interaction metaphors, enabling users to
exploit sophisticated gestural interfaces, has the potential to lead to a wide series
of music and multimedia applications beyond the musical instrument metaphor.
The stateoftheart practice mainly consists of direct and strictly causal ges-
ture/sound associations, without any dynamics or evolutionary behaviour. How-
ever, research is now shifting toward higherlevel indirect strategies: these in-
clude reasoning and decisionmaking modules related to rational and cognitive
processes, but they also need to be grounded in strong perceptual, cognitive and
emotional bases. Music theory and artistic research in general can feed SMC re-
search with further crucial issues. An example is expressive autonomy; [19] that
is, the degree of freedom an artist leaves to a performance involving an interac-
tive music system.
59

ii
Roadmap  2007/4/1  16:00  page 60  #74
i
i
i
i
i
i
Chapter 4. The State of the Art
Mobile music
Combining music and mobile technology promises exciting future developments.
Devices such as mobile phones, Walkmans and mp3 players have already resha-
ped the general music experience of listeners. With new properties such as ad
hoc networking, Internet connection and contextawareness, mobile music tech-
nology offers countless new artistic, commercial and sociocultural opportunities
for music creation, listening and sharing. Through the use of these new tech-
nologies, new forms of interaction with music lie ahead.
Weakness of the new interfaces
The possibilities offered by digital instruments and controllers are indeed end-
less. Almost anything can be done and much experimentation is going on. Yet
the fact is that there are not that many professional musicians who use them as
their main instrument. No recent electronic instrument has reached the (limited)
popularity of the Theremin or the Ondes Martenot, invented in 1920 and 1928
respectively. Successful new instruments exist, but they are not digital, not even
electronic. The most recent successful instrument is the turntable, which became
a real instrument in the early eighties when it started being played in a radically
unorthodox and unexpected manner. It has since then developed its own musical
culture, techniques and virtuosi. For the success of new digital instruments, the
continued study of sound control, mapping, ergonomics, interface design and re-
lated matters is vital. That is, what is needed is lowerlevel and focused research
which tries to solve independent parts of the problem. But clearly these studies
are insufficient and we require integral studies and approaches which consider
not only ergonomic but also psychological, philosophical and above all, musical
issues, even if these are, by definition, nonsystematic.
4.2.2 Performance Modelling and Control
A central activity in music is performance, that is, the act of interpreting, structur-
ing, and physically realising a work of music by playing a musical instrument. In
many kinds of music  particularly so in Western art music  the performing
musician acts as a kind of mediator: a mediator between musical idea and instru-
mental realisation, between written score and musical sound, between composer
and listener/audience. Music performance is a complex activity involving phys-
ical, acoustic, physiological, psychological, social and artistic issues. At the same
time, it is also a deeply human activity, relating to emotional as well as cognitive
and artistic categories.
Understanding the emotional, cognitive and also (bio)mechanical mechanisms
and constraints governing this complex human activity is a prerequisite for the
design of meaningful and useful music interfaces (see section 4.2.1) or more gen-
eral interfaces for interaction with expressive media such as sound (section 4.2.3).
The research in this field can be seen as ranging from studies aimed at understand-
60

ii
Roadmap  2007/4/1  16:00  page 62  #76
i
i
i
i
i
i
Chapter 4. The State of the Art
ductor. Several attempts have been made to control the tempo and dynamics
of a computerplayed score with some kind of gesture input device. For exam-
ple, [12] describes a method for interactively controlling, in real time, a system
of performance rules that contain models for phrasing, microlevel timing, ar-
ticulation and intonation. With such systems, highlevel expressive control, for
example of the communicated emotional content, can be achieved. Dynamically
controlled music in computer games is another important future application.
Visualisation of musical expressivity, though perhaps an unusual idea, also has a
number of useful applications. In recent years, a number of efforts have been
made in the direction of new display forms of expressive deviations in music
performance. Langner and Goebl [13] have developed a method for visualis-
ing an expressive performance in a tempoloudness space: expressive deviations
leave a trace on the computer screen in the same way that a worm does when
it wriggles over sand, producing a sort of `fingerprint' of the performance. This
and other recent methods of visualisation can be used for the development of
new multimodal interfaces for expressive communication, in which expressiv-
ity embedded in audio is converted into visual representation, facilitating new
applications in music research, music education and HCI, as well as in artistic
contexts. A visual display of expressive audio may also be desirable in environ-
ments where audio display is difficult or must be avoided, or in applications for
hearingimpaired people.
For many years, research in HumanComputer Interaction in general and in
sound and music computing in particular was devoted to the investigation of
more cognitive, abstract aspects. In the last ten years, however, a great number
of studies have emerged which focus on emotional processes and social interac-
tion in situated or ecological environments. Examples are the research on Af-
fective Computing at MIT and research on KANSEI Information Processing in
Japan. The broad concept of `expressive gesture', including music, human move-
ment and visual (e.g., computer animated) gesture, is the object of much contem-
porary research.
Key Issues
A deeper understanding of music performance
Despite some successes in computational performance modelling, current models
are extremely limited and simplistic visàvis the complex phenomenon of mu-
sical expression. It remains an intellectual and scientific challenge to probe the
limits of formal modelling and rational characterisation. Clearly, it is strictly im-
possible to arrive at complete predictive models of such complex human phe-
nomena. Nevertheless, work towards this goal can advance our understanding
and appreciation of the complexity of artistic behaviours. Understanding music
performance will require a combination of approaches and disciplines  musi-
62

ii
Roadmap  2007/4/1  16:00  page 64  #78
i
i
i
i
i
i
Chapter 4. The State of the Art
Music interaction models in multimedia applications
There will be an increasing number of products which embed possibilities for in-
teraction and expression in the rendering, manipulation and creation of music.
In current multimedia products, graphical and musical objects are mainly used to
enrich textual and visual information. Most commonly, developers focus more
on the visual rather then the musical component, the latter being used merely as
a realistic complement or comment to text and graphics. Improvements in the
humanmachine interaction field have largely been matched by improvements in
the visual component, while the paradigm of the use of music has not changed
adequately. The integration of music interaction models in the multimedia con-
text requires further investigation, so that we can understand how users can in-
teract with music in relation to other media. Two particular research issues that
need to be addressed are (1) models for the analysis and recognition of users' ex-
pressive gestures and (2) the communication of expressive content through one
or more nonverbal communication channels mixed together.
4.2.3 Sound Interaction Design
Soundbased interactive systems can be considered from several points of view
and several perspectives: content creators, producers, providers and consumers
of various kinds, all in a variety of contexts. Sound is becoming more and more
important in interaction design, in multimodal interactive systems, in novel multi-
media technologies which allow broad, scalable and customised delivery and con-
sumption of active content. In these scenarios, some relevant trends are emerg-
ing that are likely to have a deep impact on sound related scientific and techno-
logical research in the coming years. Thanks to research in Auditory Display, In-
teractive Sonification and Soundscape Design, sound is becoming an increasingly
important part of Interaction Design and HumanComputer Interaction.
Auditory Display is a relatively new field that has already reached some kind of
consolidated state. A strong community in this field has been operating for more
than twenty years (see http://www.icad.org/). Auditory Display and Sonification are
about giving audible representation to information, events and processes. Sound
design for conveying information is, thus, a crucial issue in the field of Auditory
Display. The main task of the sound designer is to find an effective mapping be-
tween the data and the auditory objects that are supposed to represent them in
a way that is perceptually and cognitively meaningful. Auditory warnings are per-
haps the only kind of auditory displays that have been thoroughly studied and
for which solid guidelines and best design practices have been formulated. A
milestone publication summarising the multifaceted contributions to this sub
discipline is the book edited by Stanton and Edworthy [5].
Interactive Sonification is a more recent field of research. Due to the rapid growth
in relevance and diffusion of interactive systems in almost all scenario of life, it
64

ii
Roadmap  2007/4/1  16:00  page 67  #81
i
i
i
i
i
i
Sound and Music Computing
poral basis, and it is also natural to expect that the active involvement of the re-
ceiver may lead to better understanding, discoveries and aesthetic involvement.
In interactive sonification, the user may play the role of the performer in music
production. In this sense, the interpreter of a precisely prescribed music score,
adding expressive nuances, or the jazz improviser jiggling here and there within a
harmonic sieve could be two good metaphors for an interactive sonification pro-
cess.
Sound and multimodality
Recently, Auditory Display research and Sonification research have also entered
the field of multimodal and multisensory interaction, exploiting the fact that
synchronisation with other sensory channels (e.g., visual, tactile) provides im-
proved feedback. An effective research approach to the kinds of problem that
this enterprise throws up s the study of sensorial substitutions. For example, a
number of sensory illusions can be used to `fool' the user via crossmodal inter-
action. This is possible because every day experience is intrinsically multimodal
and properties such as stiffness, weight, texture, curvature and material are usu-
ally determined via cues coming from more than one channel.
Soundscape Design
A soundscape is not an accidental by product of a society. On the contrary, it
is a construction, a more or less conscious `composition' of the acoustic environ-
ment in which we live. Hearing is an intimate sense similar to touch: the acous-
tic waves are a mechanical phenomenon and they `touch' our hearing apparatus.
Unlike eyes, the ears do not have lids. It is thus a delicate and extremely impor-
tant task to take care of the sounds that form the soundscape of our daily life.
However, the importance of the soundscape remains generally unrecognised and
a process of education which would lead to more widespread awareness is ex-
tremely urgently needed.
References
[1] A. Gabrielsson. Music Performance Research at the Millennium. Psychology of Music, 31(3):221
272, 2003.
[2] Gerhard Widmer and Werner Goebl. Computational Models of Expressive Music Performance:
The State of the Art. Journal of New Music Research, 33(3):203216, 2004.
[3] C. Saunders, D. Hardoon, J. Shawe-Taylor, and G. Widmer. Using String Kernels to Identify Fa-
mous Performers from their Playing Style. In Proceedings of the 15th European Conference on Machine
Learning (ECML'2004), Pisa, Italy, 2004.
[4] William W. Gaver. Auditory Display: Sonification, Audification and Auditory Interfaces, chapter Using
and Creating Auditory Icons, pages 417446. Addison Wesley, 1994.
[5] Neville A. Stanton and Judy Edworthy. Human Factors in Auditory Warnings. Ashgate, Aldershot,
UK, 1999.
67

ii
Roadmap  2007/4/6  16:00  page 72  #86
i
i
i
i
i
i
Chapter 4. The State of the Art
4.3.2 Music Generation Modelling
Due to its symbolic nature  close to the natural computation mechanisms avail-
able on digital computers  music generation was among the earliest tasks as-
signed to a computer, possibly predating any sound generation attempt (related
instead to signal processing). The first wellknown work generated by a com-
puter, Lejaren Hiller's Illiac Suite for string quartet, was created by the author (with
the help of Leonard Isaacson) in 195556 and premiered in 1957. At the time,
digital sound generation was no more than embryonic (and for that matter, ana-
log sound generation was very much in its infancy too). Since these pioneering
experiences, the computer science research field of Artificial Intelligence has
been particularly active in investigating the mechanisms of music creation.
Soon after its early beginnings, music generation modelling split into two major
research directions, embracing compositional research on one side and musico-
logical research on the other. While akin to each other, these two subdomains
pursue fundamentally different goals. In more recent times, the importance of a
third direction, mathematical research on music creation modelling, has grown
considerably, perhaps providing the necessary tools and techniques to fill in the
gap between the above disciplines.
Music generation modelling has enjoyed a wide variety of results of very differ-
ent kinds in the compositional domain. These results obviously include art mu-
sic but they certainly do not confine themselves to that realm. Research has in-
cluded algorithmic improvisation, installations and even algorithmic Muzak cre-
ation. Focusing on algorithms in music composition is an obvious choice when
contemplating the generation of music by computers. Algorithmic composition
applications can be divided into three broad modelling categories: modelling tra-
ditional compositional structures, modelling new compositional procedures, and
selecting algorithms from extramusical disciplines [8]. Some of this last type
have been used very proficiently by composers to create specific works. These
algorithms are generally related to selfsimilarity (a characteristic that is closely
related to that of thematic development, which seems to belong to many types
of music) and they range from genetic algorithms to fractal systems, from cellu-
lar automata to swarming models and coevolution. In this same category, a per-
sistent trend towards using biological data to generate compositional structures
has developed since the 60's. Using brain activity (through EEG measurements),
hormonal activity, human body dynamics and the like, there has been a constant
attempt to equate biological data with musical structures [12]. Another use of
computers for music generation has been that of computerassisted compo-
sition. In this case, computers do not generate complete scores. Rather, they
provide mediation tools to help composers manage and control some aspects of
musical creation. Such aspects may range, according to the composers' wishes,
from extremely relevant decisionmaking processes to minuscule details. While
computer assistance may be a more practical and less `generative' use of comput-
72

ii
Roadmap  2007/4/6  16:00  page 75  #89
i
i
i
i
i
i
Sound and Music Computing
[4] David Temperley. The Cognition of Basic Musical Structures. MIT Press, Cambridge, MA, 2004.
[5] Nicola Orio. Music Retrieval: A Tutorial and Review. Foundations and Trends in Information Retrieval,
1(1):190, 2006.
[6] George Tzanetakis and Perry Cook. Musical Genre Classification of Audio Signals. IEEE Trans-
actions on Speech and Audio Processing, 10(5):293302, 2002.
[7] Robert Zatorre. Music, the Food of Neuroscience?. Nature, 434:312315, 2005.
[8] Martin Supper. A few remarks on algorithmic composition. Computer Music Journal, 25(1):4853,
2001.
[9] George Papadopoulos and Geraint Wiggins. Ai methods for algorithmic composition: A survey,
a critical view and future prospects. In Proceedings of the AISB'99 Symposium on Musical Creativity,
1999.
[10] Guerino Mazzola. Mathematical Music TheoryStatus Quo 2000, 2001.
[11] Nicola Bernardini. Semiotics and Computer Music Composition. In Proceedings of the International
Computer Music Conference 1985, San Francisco, 1985. CMA.
[12] Eduardo Miranda, Ken Sharman, Kerry Kilborn, and Alexander Duncan. On Harnessing the
Electroencephalogram for the Musical Braincap. Computer Music Journal, 27(2):80102, 2003.
75

ii
Roadmap  2007/4/1  16:00  page 76  #90
i
i
i
i
i
i

ii
Roadmap  2007/4/1  16:00  page 77  #91
i
i
i
i
i
i
CHAPTER 5
Challenges and Strategies
In this chapter, after having reviewed the identity of SMC, its context and the
key issues which are the focus of current research, we can look ahead, identify
key challenges and propose the strategies with which to face them. This is the
main contribution of this Roadmap - a proposal for a pathway to the future in
the SMC field.
We have consciously taken a broad view of research in SMC. We have recog-
nised several contextual issues which have a big impact in our area. For several
reasons, these have to be taken into account when delineating the key strategies
that would help push SMC forward. First, as SMC is an multidisciplinary sub-
ject nourished by various research disciplines, there are strong mutual influences.
Second, because most SMC research is of the applied kind, an understanding of
the industrial and social contexts helps define many of the targets to be aimed
at. Finally, as SMC is a field without clear or well-established educational curric-
ula, the future of the academic framework will influence its research community.
Our pathway proposal identifies five challenges. Two of these cover research goals,
one addresses educational aspects, another focuses on knowledge transfer and
the last one is centred on social concerns. Then, to meet each challenge, we have
proposed a number of strategies.
77

ii
Roadmap  2007/4/1  16:00  page 78  #92
i
i
i
i
i
i
Chapter 5. Challenges and Strategies
5.1 CHALLENGE 1: TO DESIGN BETTER SOUND OBJECTS AND
ENVIRONMENTS
Improvements in the sounds produced by the objects present in our environment
will enhance the affective and emotional character of these objects and consequently
our quality of life.
We modern humans are constantly surrounded by sound  both natural and
artificial. Due to long habituation, we are often relatively unaware of the rich-
ness of the soundscape surrounding us and of its potential effects, not only in
terms of its information content and distraction and manipulation potential but
also its healthrelated aspects. The artefacts and devices which surround and
equip us often come with artificially designed sounds that are poorly suited to
their function and aesthetics (think of mobile phones, for instance). Due to the
widespread availability of broadcast and reproduced sound, we live in `schizo-
phonic' environments, where sound is separated from its source. In addition,
public and personal environments tend to be cluttered with unwanted sound and
music. This trend is impairing the exploitation and appreciation of sound and
music.
There are several areas where sound modelling is not yet exploited and we are
stuck with prerecorded sound (e.g. in computer interfaces). This lack impairs
the flexibility and effectiveness of communication and negatively affects the emo-
tional and affective character of objects in use.
On the music side, the notion of musical instrument is being challenged by in-
formation technology and the widespread availability of networked sensors and
actuators. Sound synthesis is definitely remains an unsolved problem but at the
same time the concepts of music instrument and of sound device are taking quite
a number of new directions.
In short, the growing abundance of artificial sounds in our environment, cou-
pled with the rapid advances in information and sensor technology, present SMC
with unprecedented research challenges, but also opportunities to contribute to
improving our audible world.
Strategies to address challenge 1:
Strategy 1: Seek directions in which to extend the notion of mu-
sical instrument.
Objects may be turned into musical instruments as soon as someone starts ex-
ploiting their expressive capabilities and employing some kind of virtuosity. This
78

ii
Roadmap  2007/4/1  16:00  page 79  #93
i
i
i
i
i
i
Sound and Music Computing
had happened with many everyday objects in the past, and, in a world of sen-
sorised and networked objects and spaces, is likely to occur in the future. Fur-
thermore, new forms of music performance based on distributed and collabo-
rative instrumentation require a research effort to make sure that the complex
environments of the future offer flexible support to creative uses (or unexpected
abuses) of technological infrastructures.
Strategy 2: Improve technologies for pervasively producing, trans-
forming and delivering sounds.
All SMC research involves, in one form or another, the manipulation of sound
signals. Research is needed to improve synthesis algorithms, both those based on
Signal/Spectral Models and those based on Physical Modelling. At a more struc-
tural level, ComputerAssisted Composition should be included and seamlessly
integrated with these algorithms. For natural human/sound interactions at an in-
dividual level, all advances in Personal Sound Reproduction should be encour-
aged. They might range from 3D audio over headphones to Computer/Brain In-
terfaces, from prophylactic uses as in cochlear implants to generaluse biofeed-
back techniques.
Strategy 3: Intensify research in sound modelling that goes be-
yond imitation towards capturing the communicative potential of
sound.
Sound is a powerful means through which to convey rapid and continuous infor-
mation about objects, events, processes, functions and relations. Research should
isolate the physical, acoustic and perceptual features that contribute to the salience
of such items, so that sounds can be molded according to specific communica-
tion needs. The result of such advances will be that information display, interac-
tion design and artistic expression may benefit from suitable sound models ac-
cessible via meaningful parameter spaces which thus go well beyond collections
of prerecorded samples.
Strategy 4: Promote research in fields involved in the shaping of
natural, artificial and cultural acoustic ecosystems.
The SMC community should enlarge its scope to give itself the potential to af-
fect fields concerned with designing spaces for a better quality of life on various
scales: product design, architecture, urban planning, landscape design and conser-
vation. Sound is increasingly perceived as an important component at all levels,
not only as a source of pollution, but as a facilitator of interaction and a compo-
nent of the aesthetic experience of a place or its genius loci.
79

ii
Roadmap  2007/4/1  16:00  page 83  #97
i
i
i
i
i
i
Sound and Music Computing
research should transcend its current (narrow) focus on mostly classical music.
This favours abstract musicscorecentred models that neglect the human in the
loop. Instead, it should put more systematic effort into studying the processes of
expression transmission in musical environments, with a focus on all three com-
ponents of the communication channel: the expressive sound (music) itself, the
performer and the listener (with the whole body  see next item).
Strategy 4: Develop an embodied, integrated approach to per-
ception and action.
There is a growing consensus in cognitive science that perception, be it natural
or artificial, cannot be fully understood without reference to action. This aware-
ness is especially important for SMC, where action is intrinsically linked to sound
interaction and music making. Research in perceptionaction topics should thus
be encouraged. Ergonomics is the most applied level of research where percep-
tion and action meet. At a more fundamental level, sensorymotor theories and
embodiment of cognitive abilities are defining and formalising the important as-
pects of the perception and action loop. These strategies will have a clear impact
on the much needed change in human/computer interfaces for SMC, which may
include wholebody interaction for expressive purposes.
Strategy 5: Intensify multimodal and multidisciplinary research
on computational methods for bridging the semantic gap in mu-
sic.
The Semantic Gap in computational music  the discrepancy between what can
be recognised in music signals by current stateoftheart methods and what hu-
man listeners associate with music  is the main obstacle on the way towards
truly intelligent and useful musical companions. Current research efforts aim at
the automatic recognition and modelling of higherlevel musical patterns (e.g.,
rhythmic or harmonic structure), but they still essentially adhere to the traditional
bottomup pattern analysis scenario. This is comparatively easy to master. But
really bridging the semantic gap will require a radical reorientation towards the
integration of topdown modelling of (incomplete) musical knowledge and ex-
pectations, and also towards a widening of the notion of musical understand-
ing by embracing and exploiting other media and modalities, including the Web.
This research will have to be notably multidisciplinary, involving, among others,
specialists in musicology, music perception, Artificial Intelligence and Machine
Learning.
83

ii
Roadmap  2007/4/1  16:00  page 86  #100
i
i
i
i
i
i
Chapter 5. Challenges and Strategies
technological innovation. In return, science provided many methodologies and
tools which were greatly inspiring for several art forms. However, the drive for
innovation coming from art has progressively diminished due to the increasing
specialisation of the domains involved. The arts can again play a creative role
when curricula in SMC are better integrated. Composition and Sound Design is
a typical example where this integration is possible. Specific proactive initiatives
must be implemented to allow composers and content creators to complement
their training in Europe and abroad. The complementarity of art and science can
be utilised through the careful design of specific courses at the Master's level and
the possibility of longterm collaboration between centres in Europe at the PhD
level.
Strategy 3: Promote crosscultural integration.
The recent surge in nonEuropean industries and global markets requires a re-
consideration of how education faces up to globalisation, multiculturalism and
crosscultural integration. In particular, the growing population of students in
Europe that come from different nonEuropean cultures and backgrounds re-
quires appropriate education and pedagogical approaches that reflect a concern
for multiculturalism. The challenge is to develop new types of educational col-
laboration with selected nonEuropean universities and research institutions, us-
ing a supportive framework that should fund scholarships, staff positions and re-
search mobility.
Strategy 4: Promote better coordination in Higher Education.
SMC research has a successful track record connected to an excellence spread
over several centres which have gained world leadership through complemen-
tarity and coordination, duly supported by EC funding mechanisms. In order
to maintain the leadership in this domain, this excellence should be exported to
the Higher Education domain. This can be achieved through the integration of
Masters' curricula, PhD programmes and postgraduate activities at the European
level. In this context, student/teacher mobility must be encouraged through ap-
propriate funding actions. Stable and enduring support for common activities
such as the Sound and Music Computing Summer School and targetoriented
Sound and Music Computing ateliers and workshops must be granted in order to
provide continuity in Higher Education.
86