StateStream: a Developer-Centric Approach
Towards Unifying Interaction Models and Architecture
Gerwin de Haan
Data Visualization and VR Group
Delft University of Technology
P.O. Box 5031, 2600 GA, Delft, The Netherlands
g.dehaan@tudelft.nl
Frits H. Post
Data Visualization and VR Group
Delft University of Technology
P.O. Box 5031, 2600 GA, Delft, The Netherlands
f.h.post@tudelft.nl
ABSTRACT
Complex and dynamic interaction behaviors in applications
such as Virtual Reality (VR) systems are difficult to de-
sign and develop. Reasons for this include the complexity
and limitations in specification models and their integration
with the underlying architecture, and lack of supporting de-
velopment tools. In this paper we present our StateStream
approach, which uses a dynamic programming language to
bridge the gap between the behavioral model descriptions,
the underlying VR architecture and customized development
tools. Whereas the dynamic language allows full flexibility,
the interaction model adds explicit structures for interac-
tive behavior. A dual modeling mechanism is used to cap-
ture both discrete and continuous interaction behavior. The
models are described and executed in the dynamic language
itself, unifying the description of interaction, its execution
and the connection with external software components.
We will highlight the main features of StateStream, and
illustrate how the tight integration of interaction model and
architecture enables a flexible and open-ended development
environment. We will demonstrate the use of StateStream
in a prototype system for studying and adapting complex
3D interaction techniques for VR.
Categories and Subject Descriptors
D.2.2 [ Software Engineering ]: Design Tools and Tech-
niques— State diagrams; user interfaces; H.5.2 [ Informa-
tion interfaces and presentation ]: User Interfaces—
Graphical user interfaces; prototyping
General Terms
Design, Algorithms
Keywords
Model-Driven Engineering, User Interface Description Lan-
guage, 3D interaction, Python
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
EICS’09, July 15–27, 2009, Pittsburgh, Pennsylvania, USA.
Copyright 2009 ACM 978-1-60558-600-7/09/07 ...$10.00.
Figure 1: Example of a dynamic, 3D manipulation
technique in a multi-user VR setup. When multiple
interaction tools can operate on multiple objects,
interaction behavior can become difficult to design
and program.
1. INTRODUCTION
The look and feel of a well designed interaction technique
appears simple, logical and intuitive to its users. Little
thought goes to the often painstaking and time-consuming
development process of making interaction techniques work
as designed, error-free and well-tuned. Even with use of
existing interface and interaction modeling methodologies,
taxonomies and software tools, it remains hard to design,
to model, to integrate, to debug and to evaluate complex
interaction techniques and scenarios. Trends in new input
modalities such as multi-user displays and distributed sys-
tems further complicate the design and development of in-
terfaces, the underlying software models, architectures and
tools.
In our recent efforts in developing multi-user and multi-
handed input 3D interaction techniques [12], we encounter
many situations where the complexity of the interaction de-
scription explodes. Simple interaction techniques are rela-
tively straightforward to design and implement, but their
re-use through variations and combinations can easily lead
to unexpected results, often only to be discovered while al-
ready applied in a VR application.
Consider the example in Figure 1, where a 3D box can be
selected and freely manipulated with a ray controlled by a
tracked stylus. This works intuitively for a single user oper-
ating the VR system, but when a second user joins in and
grabs the same object, the interaction behavior can become
13

Figure 2: Cyclic development. From an informal be-
havior description, one generates a model abstrac-
tion which closely maps to running code. Errors or
unexpected situations may occur during interaction,
and need to be detected, analyzed and fixed. State-
Stream integrates model, code and tools.
more complex. As a side-effect of our original event-based
implementation, the object’s control is just taken over by
the last selecting user. Naturally, one would want to decide
which alternative behavior is used, such as averaging inter-
acting forces, bending one user’s ray or scaling the object.
The relations between all components depend on which com-
bination of interaction tools operate on which object types.
To allow for these alternatives, one would need to rethink
and re-implement the interaction behavior in detail.
Although much effort can be put into capturing these new
situations in hard coded, imperative commands of callbacks
and event-handlers, problems concerning the number of ob-
ject relations and exceptions to the rule either get overlooked
or quickly overwhelm development tasks. In practice, classic
interaction modeling concepts and development tools often
lack integration and rarely provide the right level of abstrac-
tion for effective design and problem solving. These restric-
tions often prevent developers and designers from adopt-
ing and customizing more sophisticated 3D interaction tech-
niques in real VR applications.
Our motivation for this work follows from these issues.
We feel that (combinations of) interactive behavior are in-
herently complex and require model-based design and sup-
porting analysis tools. At the same time however, one wants
to avoid restrictions a model imposes on the flexibility of
existing software tools and existing design skills. A flexible
integration of model, architecture and the supporting tools
is important to support developers with varying skills and
backgrounds, ranging from graphics programmers to interac-
tion designers. The main problem is that many existing in-
teraction models are far separated from other, external soft-
ware components. This separation limits their descriptive
power, thereby restricting the visibility of features and is-
sues when integrated in run-time environments. This makes
it difficult to appreciate a model’s value, especially in agile
scenarios with many software components and a cyclic pro-
cess of design and development, see Figure 2. In this paper,
we address this issue of separation and discuss our developer-
centric approach. We present StateStream, a pragmatic soft-
ware approach to unify interaction models and architecture.
The contribution of StateStream is the developer-centric
approach of using a dynamic language to unify an interac-
tion model with underlying architecture and tools of inter-
active applications. The interaction model provides seman-
tic structure, but is described using in the same language
with familiar syntax as the other system components. It
allows integration and transition of existing code and con-
trol structures, but also benefit from dynamic language fea-
tures such as dynamic execution, introspection and exten-
sion at run-time. With this approach we have implemented
and integrated a dual interaction model, consisting of sep-
arated StateChart and Data Flow primitives. We describe
how this model provides powerful composition patterns for
code-reuse, how it eases integration with underlying system
components. To demonstrate the functionality of our model
and approach, we describe the creation, adaptation and re-
use of several interaction tools within our VR framework.
The remainder of this paper is organized as follows: We
first discuss related work on interaction models, tools and ar-
chitectures of current interactive graphics systems in section
2. Then, in section 3, we describe the StateStream model
and its components in technical detail. After a description of
the implementation in section 4, the process of creating sev-
eral 3D interaction techniques within our VR demonstrator
is described in more detail in section 5. Finally, we discuss
the results of this work in section 6 and conclude and give
our view on future work in section 7.
2. RELATED WORK
In this section we discuss the position of the StateStream
approach with respect to related work. We consider three
main themes of interest: model-based design, its practical
integration in the underlying architecture, and the support-
ing software tools in the development cycle.
2.1 Model-based Design
Modeling languages are considered an essential asset in de-
scribing and implementing interaction behavior while avoid-
ing detailed execution and validity issues for users. Among
the various models, the Data Flow paradigm and state based
models are the most well-known for user interaction. The
Data Flow model is widely applied to describe system flow
and interaction techniques in terms of filters, often with a
focus on reconfiguration. For example, with the InTml [9]
specification language one can describe 3D interaction tech-
niques, input devices and their connections. UNIT [22] uses
a similar model and focuses on flexible redefinition of con-
tinuous behavior of interaction techniques. The IFFI sys-
tem [23] provides an even higher abstraction to allow for
reuse of techniques across different VR toolkits. FlowVR [1]
extends the Data Flow approach over its entire VR archi-
tecture to provide distributed capabilities. Data Flow mod-
els excel in their description of continuous behavior compo-
nents, but often require ill-formed constructions to support
simple tasks such as message passing and event handling.
State-based models are better suited to model discrete
behavior and integrate with event-based systems. For spec-
ifying reactive user interface behavior, special interest goes
to StateCharts [13], which provides a visual formalism for
hierarchical, concurrent statemachines. The hierarchy and
concurrency can avoid state explosion, see [28]. Many State-
Chart variants exist with different model properties and op-
erational semantics [3, 18]. Recent examples of StateChart-
inspired user interface modeling approaches include HsmTk
[4] for 2D direct manipulation interfaces, CHASM [27] for
3D user interfaces and d.tools [14] for physical prototyping.
14