A Framework for Tangible User Interfaces
Boriana Koleva, Steve Benford, Kher Hui Ng, Tom Rodden
The University of Nottingham
School of Computer Science & IT
Nottingham, NG8 1BB, U.K.
+44 115 8466530
{bnk, sdb, khn, tar}@cs.nott.ac.uk
ABSTRACT
This paper extends our understanding of tangible user interfaces
(TUIs) by considering the different ways in which physical and
digital objects can be computationally coupled. It proposes a
framework based around the degree of coherence between
physical and digital objects. Links between physical and digital
objects are described in terms of a set of underlying properties
(transformation, sensing, configurability, lifetime, autonomy,
cardinality and link source). We use our framework to classify a
representative selection of existing TUI systems. This
classification raises key implications for the field of tangible
computing. In particular our focus on enriching physical-digital
links highlights the need to consider the asymmetry of these links,
issues surrounding their configuration and the need to represent
their nature to developers and users.
Categories and Subject Descriptors
H.5.2 [Information Interfaces and Presentation]: User
Interfaces – theory and methods.
General Terms
Design, Theory
Keywords
Tangible user interfaces, design framework, interaction models
1. INTRODUCTION
Tangible computing [11] allows users to interact directly with
computational artifacts by manipulating everyday physical objects
rather than using traditional graphical interfaces and dedicated
physical interface devices such as mice and keyboards. A variety
of systems have been developed to date that illustrate the tangible
interface principle. Some notable examples include:

The TangibleGeospace application of the metaDesk [22],
where physical representations of geographical features are
used to manipulate a digital map;

Illuminating Light [24], where physical models of optical
elements are used to create a simulated optical layout;

The Passage system [16], which provides a mechanism for
transporting digital information by linking it to physical
objects;

WebStickers [10], where everyday physical object act as
bookmarks for web pages;

The tangible tools (tongs, eraser and magnet) provided by the
Surface Drawing system [21];

Illuminating Clay [20] where users interact directly with a
clay model of a landscape and observe the effects of geometric
changes.

Storytent [8], where physical balloons are used as identifiers
for virtual balloon objects and flashlights are used as pointing
devices for manipulating them.
Just as concrete examples of tangible user interfaces (TUIs) are
proliferating, so conceptual frameworks are also emerging to help
researchers, designers and developers classify what constitutes a
TUI and to understand the various ways in which physical objects
can be combined with digital information. Like the MVC
interaction model for GUIs [4] and the PAC interaction model for
dialog design [5], these frameworks seek to highlight the main
components of TUIs.
Ullmer and Ishii have proposed the model-control-representation
(physical and digital) (MCRpd) interaction model for tangible
interfaces [23], which highlights the integration of physical
representation and control with this type of interface. Holmquist
et al have suggested a broader taxonomy of how physical and
digital objects can be coupled [10]. They propose the categories
of containers as generic objects for moving information between
devices or platforms, tokens as objects for accessing stored
information (the nature of which is physically reflected in the
token) and tools as object for manipulating digital information.
This paper aims to further extend our understanding of the
different ways in which physical and digital objects can be
computationally coupled. It introduces a framework that is based
around the idea of the “degree of coherence” between physical
and digital objects. This is further broken down into the concept
of links between physical and digital objects that are described in
terms of a set of underlying properties. We use our framework to
classify a representative selection of TUI systems. In turn, this
classification raises key implications for the field of tangible
computing.
This paper was presented at "Physical Interaction (PI03) -
Workshop on Real World User Interfaces", a workshop at the
Mobile HCI Conference 2003 in Udine (Italy). September 8,
2003. The copyright remains with the authors.
Further information and online proceedings are available at
http://www.medien.informatik.uni-muenchen.de/en/events/pi03/

2. DEGREE OF COHERENCE
The first concept that we introduce as a means of distinguishing
between different types of tangible UIs is “degree of coherence”.
It is proposed that relationships between physical and digital
objects can be rated along a coherence continuum, where the level
of coherence represents the extent to which linked physical and
digital objects might be perceived as being the same thing. That is
whether the physical and the digital artifact are seen as one
common object that exists in both the physical and the digital
domain or whether they are seen as separate but temporarily
interlinked objects. Figure 1 shows the coherence continuum
along with some proposed categories of TUI types.
Interface objects that establish the weakest level of coherence with
the computational artefacts they operate are termed “general
purpose tools”. Using such a tool a user can select to manipulate
any one of many digital objects and perform different
transformations (depending on the application). Examples include
traditional physical interface devices such as mice and joysticks.
The next category along the coherence continuum, named
“specialised tools”, encompasses interface objects that have a
more specialised function but still temporarily connect to
potentially many different digital objects. Examples include the
tongs, eraser and magnet from the Surface Drawing system [21]
and the optical instruments, such as mirrors, beam-splitters, lenses
etc., from the Illuminating Light system [24].
The “identifier” category represents interface objects which act as
bookmarks for retrieving computational artefacts. The passenger
objects in the Passage system [16] and the bar-coded everyday
objects in the WebSticker system [10] are examples of this
category. Here the physical object is a token representing a digital
artefact and the two are often more permanently coupled.
Interface objects that belong to the “proxy” category are even
more coherent with the digital objects they are coupled to. This is
because proxies are more permanently associated with, and allow
a more extensive manipulation of, their digital counterpart (more
than identification for subsequent retrieval). Examples include the
physical building models in the Geospace application of the
metaDesk [22] and the pucks on the Senseboard as used to
represent conference papers [14].
The “projection” category encompasses relationships where the
digital artefact is seen as a direct representation of some properties
of the physical object and so its existence is dependent on the
physical object. An example is the representation of human
activity in a physical foyer as a digital pattern projected on the
wall of the ambientRoom [12].
Finally we can create the illusion that two coupled objects are one
and the same if they are visible only one at a time, making smooth
transitions between the physical and the digital space. For
example a physical object may pass though a traversable interface
[15] and appear as a virtual object on the other side of the display
(in the virtual space) or the spaces can be superimposed in such a
way that all actions appear coherent (e.g. highly registered
augmented reality).
Unpacking this idea of coherence a bit further, we propose that
what distinguishes the different categories are underlying
differences in what interactions are sensed, the type of effects
mediated between the coupled objects, the duration of the
coupling, autonomy of the digital artefact and configurability of
the coupling. The following section develops this idea further,
proposing a detailed set of properties that we can use to describe
and design different types of links.
3. COHERENCE AND LINK PROPERTIES
3.1 Transformation
This property describes whether the effect mediated between
linked objects is literal or transformed. If actions are mediated
literally, movement of the physical object for example, will result
in the same movement of the digital object. This is the case with
the phicons and virtual building models in Tangible Geospace
[22] and the manipulation of the CUBIK interface [17]. Here the
shape of a physical cube is altered by pushing and pulling its sides
and these manipulations are directly mediated to a virtual cube
whose shape changes in a corresponding manner.
On the other hand, the effect between linked objects can be
transformed. For example positioning a physical object on a
predefined place may trigger an animation of a digital object
and/or for the digital object to emit sounds. Another example is
the magnet tool in the Surface Drawing system [21], which
changes the meaning of the drawing action to that of altering
existing geometry. Waving the magnet near the region of a
drawing pulls that region closer to the magnet.
3.2 Sensing of Interaction
This property describes what interactions with the interface object
and its surrounding environment are sensed and transmitted to the
destination object. This can range from detecting and responding
to the presence of the source object in a specified area [16] to
mediating manipulations in the full 6 degrees of freedom. E.g.
translations and rotations in a plane are sensed for the metaDesk
phicons [22] and for the CUBIC interface [17] scaling in the X, Y
and Z axes are transmitted. An example of sensing actions in the
surrounding environment is using video processing to detect
gestures such as pointing at the physical objects, e.g. the
DigitalDesk [26].
coherence
weak strong
General-
purpose tool Identifier
Proxy Illusion of
same objectsProjection
Specialised
Tool
Tangible interfaces
Figure 1: TUI categories along the coherence continuum

3.3 Configurability of Transformation
This property describes whether the transformation mediated
between two linked objects remains fixed for the lifetime of the
link or whether it is configurable over time. For example in the
Illuminating Light system [24] each tangible object has a fixed
transformation associated with it. E.g. a representation of an
optical-grade mirror always has the effect of reflecting the virtual
beams of light. On the other hand the pen for interacting with the
Toshiba tablet PC changes its effect from a left to a right click on
a digital objects when its button is pushed down.
3.4 Lifetime of link
This property describes for how long a physical and a digital
object remain linked. A physical object may be consecutively
linked to different digital artefacts in the lifetime of the
application. For example a flashlight in the Storytent [8] can be
used to select different balloon objects. Alternatively, the
physical and digital object may remain linked for the lifetime of
the application. This is exemplified in the Tangible Geospace
[22] where the phicons are permanently bound to their digital
counterparts. Finally, a link may retain its nature across many
applications, potentially even permanently.
3.5 Autonomy
This property describes to what extent the existence of the
destination object is reliant upon the existence of the link and the
source object. For example, a digital object may be created only as
a result of its link to a physical object. This is the case with the
balloon objects in the Storytent – a virtual balloon is created
whenever a physical balloon is brought into the tent and then it is
deleted when that physical balloon leaves the tent. The destination
object may also be a representation/projection of some of the
characteristics of the source object, e.g. the digital pattern
projected on the wall of the ambientRoom [12] reflects human
activity in a physical space. In such cases if the source object
ceases to exist, the destination object would also disappear or
become meaningless.
3.6 Cardinality of Link
This describes whether an object is linked to one or more objects.
One-to-one relationships seem to be most common. For example
in the Tangible Geospace application on the metaDesk [22] each
phicon, a small physical model of a particular building, was
bound to the digital representation of that building. However it is
also possible to link a physical object to multiple digital objects,
e.g. Passage objects [16] could have been implemented so that a
single physical “passenger” can identify a selection of digital
documents (i.e. play the role of a folder). We can describe such a
configuration by saying that a link has multiple destinations.
3.7 Link Source
So far we have only discussed cases where there is a physical
interface object that mediates transformations to a digital object.
However, it is also possible for digital objects to affect the state of
the physical world. For example, haptic interfaces such as the
PHANToM [18] provide a tangible feedback to the person
manipulating digital objects and ambient displays such as Natalie
Jeremijenko’s Live Wire [25] provide tangible feedback of
activities in digital space (Ethernet traffic in this case) through
physical motion, sound and touch.
The link source property describes whether the source of the
effect is the physical or the digital object.
4. REVIEWING CURRENT SYSTEMS
We now use our proposed link properties to classify current TUIs.
First, we use the link source property to broadly divide systems
into those where the source is a physical object and those where
the source is a digital object. Thus Table 1 summarises the
properties of the example TUI systems that have been discussed
for far in which physical objects control digital ones. Table 2, on
the other hand, introduces systems where the source is a digital
object that has an effect in physical space. The examples in both
tables are broadly listed in order of increasing coherence.
As object relationships with a cardinality of one to many are rare
in current systems, this property has been omitted from the tables
and all examples illustrate links where a single physical object is
coupled to a single digital artefact.
5. IMPLICATIONS FOR TUIs
Our framework is based on unpacking the nature of the links
between tangible and digital objects and using this to classify
TUIs. This represents a shift in focus from many of the current
perspectives. Not unsurprising much of the existing work on
tangible interfaces has tended to focus on realising the physical
artefacts associated with tangible interfaces. Current frameworks
such as those proposed by Ullmer and Ishii [23] and Holmquist
[11] while based on a connection between physical and digital
tend to leave the nature of this connection as implicit with little
reflection on the different ways in which this connection may be
manifest. Understanding TUIs based on the richness of potential
links between the digital and the physical provides us with a
slightly different perspective on their nature and outlines a
number of significant future research directions. In this section we
briefly reflect on three initial examples by considering the
asymmetry of the links, the configuration of the links involved
and the need for users and developers to understand the nature of
the link between the physical and the digital.
5.1 Tangibles that Push Back
Comparing tables 1 and 2 reveals a significant asymmetry in how
the physical and digital are linked. While there are many examples
of using physical objects to control digital objects, tangible
interfaces that react to changes in digital information are relatively
rare – there are few examples of tangible interfaces that “push
back”. It is therefore a challenge to develop techniques that will
allow us to create digital artefacts that will push back on the
physical space. These will be useful for providing tactile
information, maintaining synchronisation between digital and
physical objects and showing or monitoring digital activity
through physical movement.
Examples of push back technologies from related fields include
haptic devices for virtual reality such as the PHANToM [18],
tangible interfaces for remote collaboration such as the PsyBench
[3], and also ambient displays such as Pinwheels [6] and Table
Fountain [7]. However, it remains a challenging problem as to
how to push back through everyday objects such as blocks on a
table or post-it notes on a board. Promising approaches include
the use of airflow, waterflow, electromagnets and actuator arrays
[13].

5.2 Mobility, Reconfiguration and TUIs
Many of the examples of tangible interfaces have tended to be
based on stable arrangements between the physical and digital.
TUIs such as the MetaDesk [22] have tended to be constructed as
installations to be experienced by users as stand alone
applications. However, TUIs have also become closely associated
with Ubiquitous computing and examples such as the
ambientROOM [12] outline the potential of TUIs and
demonstrate how the digital may be physically manifest. However,
less consideration has been given to the ubiquity of information
and what happens when the physical element of TUI moves from
one context into another. The main use of mobility of physical
object has been to act as a token to access digital data.
How do the physical devices within the Ambient Room act when
they are placed within a second room? Do their existing
connections with the digital material in one ambient room persist
into the second room offering remote availability or are new links
established reflecting different digital effects? Considering the
lifetime, autonomy and configurability within the potential links
allow us to chart and understand this design space and consider
how we may support the mobility of TUIs.
5.3 Understanding the effects of TUIs
Our turn to reasoning about the links between the digital and the
physical within TUIs also seeks to develop a richer understanding
of the interactive nature of TUI. As Bellotti et al argue existing
work on sensed environments have tended to not provide
mechanisms to allow users to make sense of the interaction [2].
Essentially, as we establish richer forms of links between the
physical and the digital we need to carefully consider how the
variability inherent in these different links are conveyed to users
and how they might make sense of their interactions with TUIs.
This issue is manifest both within toolkits to realise tangible user
interfaces and how TUIs present themselves to users. Toolkits
such as Phidgets [9] provide a rich set of physical objects linked
to digital objects. The connection between the real and physical is
manifest through only one mechanism. Few structures are given to
manage a variety of forms of link. The iStuff toolkit [1] exploits
different types of events within an event heap to allow a richer set
of connections to be established. However, it is unclear which of
these connections are desirable and how these should be
structured. We would suggest our framework offers a way of
exploring this design space.
In order for the link between the physical and the digital to be
intelligible to the user we must carefully consider how these
effects are conveyed to users. What feedback is provided? How
might users understand the extent of physical manipulation? How
might the properties of the link be presented to users and how
these properties might be explored? Previous work has considered
how the properties associated with boundaries between real and
virtual environments might be presented to users [15] and we
would suggest similar explorations are needed for TUIs.
Category Example Transf. Scope ofInteraction Config. Lifetime
Autonom
y
mouse Transformed Translations in X-Z plane Configurable Temporary AutonomousGeneral
purpose tool
Tablet pen Transformed Drag, tap, tap with button pressed Configurable Temporary Autonomous
tongs, eraser,
magnet Transformed Translations in X-Z plane Fixed Temporary Autonomous
Storytent
torches Literal (ish) Translations in X-Y plane Fixed Temporary Autonomous
Specialised
Tool
Illuminating
Light Literal
Translations in X-Z plane and
rotations Y Fixed Temporary Autonomous
Passage Literal Presence Fixed Semi perm. Autonomous
WebStickers Literal Presence Fixed Permanent AutonomousIdentifier
Storytent
ballons Literal Presence Fixed Permanent Dependent
metaDesk
phicons Literal
Translations in XZ plane and
rotations around Y Fixed Permanent AutonomousProxy
CUBIC Literal Scaling in X, Y and Z axis Fixed Permanent Autonomous
Projection Display in
ambientRoom Transformed
Human
movement Fixed Permanent Dependent
Illusion of
same object
Traversable
interface Literal
Crossing
boundary Fixed Permanent Dependent
Table 1: Classification of systems with links with a physical source
Category Example Transf. Scope of Interaction Config. Lifetime Autonomy
Proxy PSyBench
objects Literal Translations in X-Z plane Fixed Permanent Autonomous
Projection PinwheelsLiveWire Transformed LAN traffic Fixed Permanent Dependent
Illusion of
same object
Traversable
interface Literal
Crossing boundary between virtual
and physical space Fixed Permanent Dependent
Table 2: Classification of systems with links with a digital source

6. CONCLUSIONS
We have proposed a framework for TUIs based around the idea of
the degree of coherence between physical and digital objects. This
was further broken down into the concept of links between
physical and digital objects that are described in terms of a set of
underlying properties. We used this proposed framework to
classify current TUIs, which raised a number of broad
implications for the field of tangible interaction. The focus on
enriching the link between physical and digital highlighted the
need to consider the asymmetry of these links, issues surrounding
the configuration of these links and the need to represent the
nature of these links to developers and users. We suggest that
these areas represent potentially fruitful directions for future
research.
7. REFERENCES
[1] Anderson Ballagas, R., Ringel, M, Stone, M., Borchers, J
(2003), “iStuff: A Physical User Interface Toolkit for
Ubiquitous Computing Environments”, CHI 2003
[2] Bellotti V, Back M, W. Edwards K, Grinter R.E., Henderson
A. (2002), Lopes C “Making sense of sensing systems: five
questions for designers and researchers” Proceedings of the
SIGCHI conference on Human factors in computing systems:
2002, ACM Press, pp 415-422.
[3] Brave, S., Ishii, H., and Dahley, A. (1998), Tangible
Interfaces for Remote Collaboration and Communication,
Proc. CSCW ’98, Seattle, Washington, USA, ACM Press,
pp. 169 – 178
[4] Burbeck, S. (1987), Applications Programming in Smalltalk-
80: How to use Model-View-Controller, http://st-
www.cs.uiuc.edu/users/smarch/st-docs/mvc.html
[5] Coutaz, J. (1987), PAC, an Object Oriented Model for
Dialog Design, Proc. Interact’87, pp. 431-436
[6] Dahley, A., Wisneski, C. and Ishii, H. (1998). Water Lamp
and Pinwheels: Ambient Projection of Digital Information
into Architectural Space, Proc. CHI'98, ACM Press, May
1998, pp. 464-471
[7] Gellersen, H.W., and Schmidt, A. (2002), Look who’s
visiting: supporting visitor awareness in the web, IJHCS,
56(1), January 2002, pp. 25 - 46
[8] Green, J., Schnädelbach, H., Koleva, B., Benford, S.,
Pridmore, T. & Medina, K. (2002), Camping in the Digital
wilderness: tents and flashlights as interfaces to virtual
worlds, Proc. CHI‘02, Minneapolis, Minnesota, ACM Press,
pp. 780-781
[9] Greenberg S, Fitchett C (2001) “Phidgets: easy development
of physical interfaces through physical widgets”, Proceedings
of ACM UIST 2001, pp 209-218
[10]Holmquist, L., Redström, J. & Ljungstrand, P. (1999),
Token-Based Access to Digital Information, Proc. HUC’99,
Karlsruhe
[11] Ishii, H. and Ullmer, B. (1997): “Tangible Bits: Towards
Seamless Interfaces between People, Bits and Atoms”, Proc.
CHI’97, ACM Press, pp. 234-241
[12] Ishii, H., Wisneski, C., Brave, S., Dahley, A., Gorbert, M.,
Ullmer, B. & Yarin, P. (1998), ambientROOM: Integrating
Ambient Media with Architectural Space, Proc. CHI'98,
ACM Press, pp. 173-174
[13] Iwata, H., Yano, H., Nakaizumi, F. and Kawamura R.
(2001), Project FEELEX: Adding Haptic Surface to
Graphics, ACM SIGGRAPH 2001, ACM press, pp. 469 –
475
[14]Jacob, R., Ishii, H., Pangaro, G. & Patten, J. (2002), A
Tangible Interface for Organising Information Using a Grid”,
Proc. CHI’02, ACM Press, pp. 339-346
[15]Koleva, B. N., Schnädelbach, H. M., Benford, S. D. &
Greenhalgh, C. (2000), Traversable interfaces between real
and virtual worlds, Proc CHI 2000, Hague, Netherlands,
ACM Press, pp.233-240
[16]Konomi, S., Müller-Tomfelde, C. & Streitz, N. (1999),
Passage: Physical Transportation of Digital Information in
Cooperative Buildings, Proc. CoBuild’99, pp. 45-54
[17]Lertsithichai & Seegmiller, (2002), Cubic interface, Proc..
CHI’02, 20 – 25 April 2002, Minneapolis, Minnesota, ACM
Press, pp.
[18]Massie, T. & Salisbury, J. (1994), The PHANToM haptic
interface: a device for probing virtual objects, Dynamic
Systems and Control, pp. 295-301
[19]Patten, J., Ishii, H., Hines, J. & Pangaro, G. (2001),
Sensetable: A Wireless Object Tracking Platform for
Tangible User Interfaces, Proc. CHI’01, Seattle, WA, ACM
Press, pp.253-260
[20]Piper, B., Ratti, C. and Ishii, H. (2002), Illuminating Clay: a
3-D Tangible Interface for landscape analysis, Proc. CHI’02,
20 – 25 April 2002, Minneapolis, Minnesota, ACM Press,
pp. 355 - 362
[21]Schkolne, S., Pruett, M. & Schröder, P. (2001), Surface
Drawing: Creating Organic 3D Shapes with the Hand and
Tangible Tools, Proc. CHI’01, Seattle, WA, ACM Press, pp.
261-267
[22]Ullmer, B. & Ishii, H. (1997), The metaDESK: Models and
Prototypes for Tangible User Interfaces, Proc. UIST ’97,
ACM press, pp. 223-232
[23]Ullmer, B. & Ishii, H. (2001), Emerging Frameworks for
Tangible User Interfaces”, in J. Carrol (ed) Human Computer
Interaction in the New Millennium, Addison-Wesley, pp.
579-601
[24]Underkoffler, J. & Ishii, H. (1998), Illuminating Light: An
Optical Design Tool with a Luminous-Tangible Interface,
Proc. CHI'98, ACM Press, pp. 464-471
[25]Weiser, M & Brown, J.S. (1995), Designing Calm
Technology
[26]Wellner, P. (1993), Interacting with Paper on the
DigitalDesk, Communications of the ACM, Vol. 36, No 7,
July 1993, pp. 87-96