PARAMETER CONTROLLED REMOTE PERFORMANCE (PCRP):
PLAYING TOGETHER DESPITE HIGH DELAY
Aristotelis Hadjakos Erwin Aitenbichler Max Mühlhäuser
TU Darmstadt
Department of Computer Science
ABSTRACT
PCRP (Parameter Controlled Remote Performance) is a
new method to play music together on the Internet
despite high delay. In many musical forms, the score of
the piece is known beforehand. PCRP measures the
deviations of the players from the given score and
describes them as parameters. The parameters, which
are transmitted instead of the audio data itself, are used
on the opposite side of the network to control the
synthesis of the score. We implemented a PCRP system
for piano duo. The system analyzes and transmits
tempo, dynamics, and articulation parameters to assure
synchronization of the players and give the users
control over tempo, dynamics, and articulation. An
evaluation of the system was performed with pianists.
The evaluation showed that musical interaction with
PCRP is possible at higher latencies than related
approaches.
1. INTRODUCTION
When playing music over the Internet, the musicians'
signals are delivered with delays. This is due to
properties of the current network technology and
physical limitations (speed of light).
Effects of delay on musical interaction have been
studied [3, 4, 6]. In an experiment conducted by Chafe
et al., two players, who were separated by artificial
delays, clapped simple rhythms [3]. The analysis of the
recorded rhythms showed that the players tended to
slow down at latencies higher than 11.5 ms. This effect
increased at higher latencies. A similar setup was used
to determine the effects of delay on piano duo
performance in an experiment by Chew et al. [4]. The
duo played different pieces at different artificially
introduced delays. The pianists reported that
satisfactory interplay was possible at delays of less than
50 ms. In a following evaluation of the recorded data
[6], the tempo variation of the players was determined.
At latencies slightly above 50 ms, the players used
compensation strategies, which led to an increased
variation of tempo. At very high latencies, the players
played in a overly steady tempo, because the
compensation strategies failed.
In this paper, we present PCRP, a new method that
enables a traditional musical interaction in presence of
high delays.
The paper is structured as follows. First, related
Network Music Performance systems are discussed
(Section 2). In Section 3, PCRP is introduced and the
analysis-transmission-synthesis process is explained in
detail. This provides the ground for the following
evaluation in Section 4. In Section 5, the offered
possibilities and limitations are discussed. Conclusions
are drawn in Section 6, and future work is described in
Section 7.
2. RELATED WORK
NMP (Network Music Performance) systems enable
geographically separated users to musically interact
over a network. Two properties of NMP systems are
especially important for the following discussion: (1)
the type of the musical interaction and (2) the
maximum delay that can be compensated (Figure 1).
NMP systems can change the type of musical
interaction to allow interaction at higher delays.
Systems without delay compensation transmit audio
data over the network and limit delay by minimizing
processing time of the application. Early systems were
TransMIDI [9] and RMCP [10]. Recent research
focuses on low-latency compression techniques [2] and
the use of advanced I/O and network hardware to
achieve immersive experiences [4, 5, 6, 7]. Systems
without delay compensation are only suitable when
network delay is low.
Delay compensation systems modify the musical
interaction so that the musicians can compensate higher
delays. In the reminder of this section different NMP
systems with delay compensation will be discussed.
One class of delay compensation systems artificially
delay the audio signal of a player to her own
headphones. The signal is delayed the same amount as
the signal needs to travel to the opposite partner
through the network. This method, which will be called
Artificial Delay method henceforth, is best suited for
instruments that can be muted like the electronic
keyboard or the electronic guitar. The Artificial Delay
method was developed by Chew et al. and evaluated
with a piano duo [5]. The piano duo was able to
compensate more delay (ca. 15 ms more) than in a
Figure 1. Properties of NMP systems

setting without delay compensation. The commercial
product eJamming [8] is a commercial implementation
of the Artificial Delay method. PCRP is a different
method for delay compensation that enables the players
to musically interact at higher delays than it is possible
with Artificial Delay systems.
Intervallic Jamming systems artificially augment the
delay between the players to a multiple of a musical
time interval, e.g., a beat, a bar, or a 12-bar blues
progression. Examples of intervallic systems are the
Global Delayed Music system [11], the M. A. S system
[13], and the Ninjam system [12]. PCRP is a different
method for delay compensation that, in contrast to
Intervallic Jamming, keeps the impression of traditional
interplay.
The TablaNet system [14] uses pattern recognition
and music prediction to enable musicians playing the
tabla, an Indian drum, to play together despite high
delay. TablaNet identifies rhythmic patterns of the
players and maps them on to known musical structures.
Symbolic representations of these musical structures are
transmitted and are used to create a similar audio output
at the receiver. The TablaNet system is specific to
Indian tabla music and can not be used by players who
want to perform a fixed score. PCRP enables players
who want to perform a fixed score to play together on
the Internet despite high delay.
3. PCRP
PCRP (Parameter Controlled Remote Performance) is
based on an analysis-transmission-synthesis scheme.
Deviations of the players from a given score are
measured and mapped to parameters. The parameters
are transmitted through the network instead of the audio
data itself. Based on the transmitted parameters, the
music is synthesized on the opposite side. The players
are hearing each other mediated by the analysis-
transmission-synthesis connection. Therefore, the users
do not feel the delay of the network. The delay of the
audio signal is exchanged with the delay of the
parameter signal. If a player changes her style of play,
the opposite player will perceive that change of style
after the correspondent pa rameters have been received.
This is less critical than a delay on the audio signal.
For the following discussion we assume a scenario
where two players, Alice and Bob, want to use PCRP to
play together. A recording of Alice’s part is also present
on Bob’s side and vice versa (Figure 2). The computer
analyzes Alice’s play and extracts parameters that
describe her deviation from the score. On Bob’s side,
the playback of Alice’s score is modified according to
the transmitted parameters. The same procedure is
applied to Bob’s play.
The parameters that are extracted by our
implementation of the PCRP method are: tempo,
dynamics, and articulation.
3.1. Architecture
We implemented a system for piano duo based on the
PCRP method. Two Casio CDP-100 keyboards are used
by the players and provide MIDI input to the system.
The system is composed of two analysis-transmission-
synthesis connections. The main components of a
connection are a score follower module, a tempo
module, a dynamics module, an articulation module,
and a synthesis module (Figure 3). The tempo,
dynamics, and articulation modules transmit parameters
to the synthesis module over the Internet.
The MIDI input of the player is passed on to the
score follower, the dynamics module, and the
articulation module. The score follower module uses
the polyphonic dynamic matching algorithm [1], which
allows to use polyphonic scores, to match the MIDI
input to the score. The score follower module generates
a sequence of timestamped score positions, which are
used by the tempo module to generate the tempo
parameter.
The dynamics and articulation modules get the
current score position from the score follower and
calculate the dynamics and articulation parameters.
This is done by comparing the dynamics and
articulation of the currently played note with the
dynamics and articulation of the corresponding score
segment. Finally, tempo, dynamics, and articulation
parameters are transmitted over the network to the
synthesis module.
3.2. Tempo Parameter
The tempo parameter enables Alice and Bob to stay
synchronized and influence each other‘s tempo.
Figure 2. Overview of PCRP