Network-Integrated Multimedia Middleware (NMM)
Marco Lohse
Motama GmbH
Saarbrücken
Germany
mlohse@motama.com
Florian Winter
Motama GmbH
Saarbrücken
Germany
fw@motama.com
Michael Repplinger
Computer Graphics Lab
Saarland University
Saarbrücken, Germany
repplinger@cs.uni-sb.de
Philipp Slusallek
Computer Graphics Lab
Saarland University
Saarbrücken, Germany
slusallek@cs.uni-sb.de
ABSTRACT
Today’s multimedia infrastructures adopt a centralized ap-
proach, where all multimedia processing takes place within
a single system. The network is, at best, used for streaming
data transmission. Since there is a strong trend towards net-
worked systems, these traditional approaches are becoming
obsolete. In contrast, the Network-Integrated Multimedia
Middleware (NMM) offers a multimedia architecture, which
considers the network as an integral part and enables the
intelligent use of devices distributed across the network.
Categories and Subject Descriptors
C.2.4 [Computer Systems Organization]: COMPUTER-
COMMUNICATION NETWORKS—Distributed Systems
General Terms
Design
Keywords
Multimedia, middleware, streaming, home networking
1. INTRODUCTION
Besides the PC, an increasing number of multimedia de-
vices – such as set-top boxes, PDAs, and mobile phones
– already provide networking capabilities. However, today’s
multimedia infrastructures adopt a centralized approach: All
multimedia processing takes place within a single system and
the network is only used for streaming predefined content
from a server to clients.
Conceptually, such approaches consist of two isolated ap-
plications, a server and a client (see Figure 1). The real-
ization of advanced scenarios is therefore complicated and
Copyright is held by the author/owner(s).
MM’08, October 26–31, 2008, Vancouver, British Columbia, Canada.
ACM 978-1-60558-303-7/08/10.
Client
Architecture
Operating
system
Server
Architecture
Operating
system
Network
System 1 System 2
Figure 1: Client/server streaming consists of two
isolated applications that do not provide fine-
grained control or extensibility.
error-prone – especially since the client has typically no or
only limited control of the server, and vice versa.
The Network-Integrated Multimedia Middleware (NMM)
presented in this paper overcomes these limitations by en-
abling access to all resources within the network [1, 2]: dis-
tributed multimedia devices and software components can
be transparently controlled and integrated into an applica-
tion. In contrast to all other multimedia architectures avail-
able, NMM is a true middleware, i.e. a distributed software
layer running in between distributed systems and applica-
tions (see Figure 2).
Operating
system
Application
Middleware
Operating
system
Network
System 1 System 2
Figure 2: A multimedia middleware is a distributed
software layer that eases application development by
providing transparency.
The unified architecture of NMM offers a simple and easy
to use interface for applications to integrate multimedia func-
tionality. Therefore, it can be used as enabling technology
for locally operating multimedia applications, but more im-

portantly for all kinds of networked and distributed multi-
media systems – spanning from embedded and mobile sys-
tems, to PCs, to large-scale computing clusters.
NMM is implemented in C++ and NMM-IDL, an inter-
face definition language. For developing new plug-ins or
applications on top of NMM, you need a set of fundamental
and optional NMM libraries. The NMM Software Developer
Kit (NMM-SDK) available online [9] greatly simplifies this
process.
NMM runs on a large number of operating systems, such
as Linux, Windows XP/Vista, and Mac OS X, and supports
a large number of platforms, such as 32/64 bit and little/big
endian CPUs, Cell processor of Playstation 3, embedded sys-
tems, etc. Being a true middleware, NMM allows to create
multimedia applications that span across all possible com-
binations of all supported operating systems and platforms.
The NMM project started in 2000. It is a joint research
and development effort of Motama, a spin-off company of
Saarland University, Germany, and the Computer Graphics
Lab at Saarland University [9]. NMM is released under a
dual-license model. While the result of our work is available
as Open Source under the terms and conditions of the GPL,
commercial licenses are offered by Motama.
2. GENERAL DESIGN APPROACH
The general design approach of the NMM architecture is
similar to other multimedia architectures – but is extended
to a truly network transparent approach as described in the
following.
2.1 Nodes, Jacks, and Flow Graphs
Within NMM, all hardware devices (e.g. a TV board) and
software components (e.g. decoders) are represented by so
called nodes. A node has properties that include its input
and output ports, called jacks, together with their supported
multimedia formats. A format precisely defines the multi-
media stream provided by specifying a type, such as “au-
dio/raw” for uncompressed audio streams, plus additional
parameters, such as the sampling rate.
Depending on the specific kind of a node, its innermost
loop produces data, performs a certain operation on the
data, or consumes data. Our system distinguishes between
different types of nodes: source, sink, converter, filter, mul-
tiplexer, demultiplexer, and a generic mux -demux node.
These nodes can be connected to create a flow graph,
where every two connected jacks need to support a “match-
ing” format. The structure of this graph then specifies the
operation to be performed, e.g. reading, decoding, and play-
back of an MP3 file.
Together, more than 60 nodes are already available for
NMM, which allows for integrating various input and output
devices, codecs, or specific filters into an application. A
complete list of available nodes is available online [9].
2.2 Messaging System
The NMM architecture uses a uniform messaging system
for all communication. There are two types of messages.
On the one hand, multimedia data is placed into buffers.
On the other hand, events contain control information, such
as a change of speaker volume. Events are identified by a
name and can include arbitrary typed parameters.
There are two different types of interaction paradigms
used within NMM. First, messages are streamed one-way
along connected jacks. This type of interaction is called in-
stream and is most often performed in downstream direction,
i.e. from sources to sinks; but NMM also allows for sending
messages in upstream direction.
Notice that both buffers and events can be sent instream.
For instream communication, so called composite events are
used that internally contain a number of events to be han-
dled within a single step of execution. Instream events are
very important for multimedia flow graphs. For example,
the end of a stream (e.g. the end of a file) can be signaled by
inserting a specific event at the end of a stream of buffers.
External listener objects can be registered to be notified
when certain events occur at a node (e.g. for updating the
GUI upon the end of a file or for selecting a new file).
Events are also employed for the second type of interac-
tion called out-of-band, i.e. request-reply interaction between
the application and NMM objects, such as nodes or jacks.
Events are used to control objects or for sending notifica-
tions from objects to registered listeners.
2.3 Interfaces
In addition to manually sending events, object-oriented
interfaces allow to control objects by simply invoking meth-
ods, which is more type-safe and convenient then sending
events. These interfaces are described in NMM Interface
Definition Language, which is similar to CORBA-IDL, but
was extended to allow for specifying out-of-band as well as
instream interaction.
For each description, an IDL compiler creates an interface
and implementation class in C++. While an implemen-
tation class is used for implementing specific functionality
within a node, an interface class is used within applications
for interacting with objects, or within nodes for creating
instream events.
2.4 Distributed Flow Graphs
What is special about NMM is the fact that NMM flow
graphs can be distributed across the network: local and
remote multimedia devices or software components encap-
sulated within nodes can be controlled and integrated into
a common multimedia processing flow graph, a distributed
flow graph. While this distribution is transparent for appli-
cation developers, no overhead is added to all locally oper-
ating parts of the graph: In such cases, references to already
allocated messages are simply forwarded – no networking is
performed at all.
For all distributed parts of a flow graph, suitable network-
ing technologies, protocols, and serialization formats are ei-
ther negotiated automatically or set up manually (i.e. scal-
able transparency [1]). While out-of-band communication
requires reliable handling of events, instream interaction can
use arbitrary combinations of protocols especially suited for
the parallel yet ordered transmission of time-sensitive multi-
media data and control events [10]. This, for example, allows
employing RTP/RTCP for multimedia data in parallel with
TCP for instream events.
Figure 3 shows an example for a distributed flow graph for
playing back MP3 files. A source node for reading data from
the local file system (GenericReadNode) is connected to a
node for decoding MPEG audio streams (MPEGAudioDe-
codeNode). This decoder is connected to a sink node for
rendering uncompressed audio using a sound board (Play-
backNode). Once the graph is started, the source nodes