R. Meersman, P. Herrero, and T. Dillon (Eds.): OTM 2009 Workshops, LNCS 5872, pp. 118–127, 2009.
© Springer-Verlag Berlin Heidelberg 2009
A Framework for Context-Aware Adaptation
in Public Displays
Jorge C.S. Cardoso1,2 and Rui José1
1
DSI, Universidade do Minho, Campus de Azurém,
4800-058 Guimarães, Portugal
2
E.Artes / CITAR, Universidade Católica Portuguesa (UCP),
4169-005 Porto, Portugal
jorgecardoso@ieee.org, rui@dsi.uminho.pt
Abstract. Several approaches for context-aware public display systems exist
but none has been able to bridge the gap between the myriad of possible
interactive features of a display and adaptation rules for its content. In this
paper, we propose a framework of digital footprints generated by the interaction
with public displays that can be used as a means to dynamically characterise a
place. We describe these footprints, how they can be generated and how they
can be used by context-aware display systems to adapt to the social
environment of a place.
Keywords: situated displays, digital footprints, context-aware.
1 Introduction
The overall idea of a context-aware public display that is able to deliver “the right
information at the right time” has been pursued for some time, but remains to be
realised. Most public displays are not even sensitive to their context. Content is
entirely defined and fully controlled by the display owner, who, at best, uses some
knowledge about the local place and the intended audience to define what might be
interesting content. This, however, is a limited approach because public places are
inherently very dynamic and diverse, supporting a broad range of situated practices. If
all the decisions must be made a priori, they will not take into account the fluidity and
heterogeneity of the social context around the display. The absence of interactive or
sensing features also means that there will be no meaningful traces of user activity:
their intended use is simply to be seen by people, so it will normally be used without
generating any information about how it was used.
Enriching public displays with interaction capabilities provide the obvious path for
addressing these two issues. Displays that offer people the possibility to interact can
lead to stronger user engagement and possibly user-generated content. They will also
be able to produce traces of user activity upon which multiple adaptation processes
can be implemented. Multiple variants of this approach have been tried to explore the
obvious potential of interactive features in generating activity traces that may support
context-aware adaptation. However, success has also been very limited, especially in
obtaining results that could be generalized to multiple adaptation processes. Part of

A Framework for Context-Aware Adaptation in Public Displays 119

the problem may originate from the clear gap between the information generated from
interaction events in public displays and adaptation processes. The key problem is that
given the broad diversity of interaction modalities and adaptation rules, there is
nothing as obvious as a user click that we can immediately take as a concept for
linking these two sides.
In this work, we propose a framework for designing context-aware public displays.
Our goal is to create a design space that can serve as tool for informing designers of
situated displays about the relation between the supported interaction modes, the type
of digital footprints they can generate and the type of adaptation processes they may
support. We started by analyzing multiple interaction alternatives from the
perspective of the information they generate. Rather than considering the specific
affordances or semantics of the interactive features offered by the display, we focused
on the type of digital trace they generate. We use the concept of digital footprint to
refer to the digital traces generated as a side-effect of implicit or explicit interactions
with the display, which can be of many different types e.g. keywords, content,
presence, indication of external content, feedback on presented content, external
personal profiles, or others. Based on their key properties, we aggregated those digital
footprints according to 4 main categories: presence, presence self-exposure, content
suggestion and actionables, providing a mapping between multiple interaction
alternatives and their contribution to the generation of local digital footprints. We then
analyse the types of adaptation processes that can be associated with each of those
digital footprints, thus providing a mapping from footprints into context-aware
adaptation processes. Overall, these mappings provide the framework for reflecting
on context-aware behaviours without being caught by the specificities of any
particular interaction or sensing mechanisms, thus providing a path for generic
context-aware mechanisms.
2 A Framework for Digital Footprints in Public Displays
The potential existence of a very broad range of sensing and interaction mechanisms,
with very diverse properties in terms of the digital footprints they can generate
represents a major challenge towards a generic model of context-aware displays. To
address this issue, we will now analyze the various types of footprint from the
perspective of their key properties. This classification is clearly display-centred, in
that the categories were defined according to the type of footprint that gets generated
at the display, without any consideration for the particular interaction model provided
to the user. We have divided the digital footprints into four categories: presence,
presence self-exposure, content suggestion and actionables.
2.1 Presence
Presence corresponds to the ability of the display to collect information about nearby
people. There are several levels of presence information that may generate very
different digital footprints, more specifically, we consider the following levels:
presence detection, presence characterisation and presence identification.

120 J.C.S. Cardoso and R. José

Presence detection
Presence detection is the most basic level of presence information in which the system
is simply able to detect whether or not there is someone nearby. Knowing that
someone is near a display, even without knowing who or how many, may be used as a
simple way to characterise a place, but is most likely to serve as a trigger for specific
content on the display to get people’s attention and attract them to interact.
Commercial motion or distance sensors can be used for this purpose. In Virtual
Kitchen [7], for example, a passive infrared sensor to detect presence in the Kitchen.
Presence was used to disable the outside “Off” button that stopped the video streaming
if someone wanted more privacy (the button was only available if the no one was in the
kitchen already). Distance can also be used by the display. There is usually a strong
correlation between distance and awareness level towards the display. In [9], for
example, an infrared distance sensor was used to determine the distance of people using
a whiteboard application and trigger different interaction modes. Computer vision
techniques such as frame differencing to determine movement can also be used for this
purpose. In the Aware Community Portals [21] frame differencing was used to detect
passers-by and triggered the display to cycle through images of recent stories. Pressure
mats [5], usually designed for security applications, can also be used as a presence
detection mechanism, for very well-defined and small areas.
The digital footprint generated by these presence detection mechanisms is a
presence/absence pattern that may help to characterise the nature of the place in terms
of people flow.
Presence characterisation
The second level of presence information is the ability to characterise presence. This
may involve determining how many people are near the display or inferring some
type of characteristic about viewers, such as age or gender. Periods of high activity or
low activity in a place, or the presence of people with specific characteristics, can all
be used to trigger specific content in the display.
Commercial people counters [25] that count the number of people entering/exiting
a room can be used by a display system to estimate the number of people nearby.
Computer vision techniques such as face detection, gender classification [23] and age
classification [10], used by some audience metering products [18], can also be used to
characterise and estimate the number of people in front of a display. These audience
metering products can deliver reports about the number, attention span, gender and
age of the viewers of a particular display.
Presence characterisation generates a richer description of people flow. The display
system is able not only to determine periods of presence/absence, it also becomes able
to characterise the changes in the number and type of viewers.
Presence Identification
Presence identification corresponds to the ability to detect unique identities in the
presences. Determining who is present, in the sense that the display system is able to
determine that the same person is present in different occasions, gives the display
system, not only the possibility to determine how many people are present, but also to
establish a correlation between different people or groups of people. This may be
achieved through face recognition techniques, but the most common approach is by

A Framework for Context-Aware Adaptation in Public Displays 121

far the use of some personal device (with Bluetooth or RFID capabilities, for
example) as a proxy for the person.
Bluetooth has been used extensively as presence detection mechanism since many
people already own a Bluetooth enabled mobile phone. The BluScreen system [20]
uses Bluetooth detection to avoid showing advertisements to users more than once.
The Cityware project [12] explored several ways in which to analyse Bluetooth
mobility traces, including a set of in situ visualizations about Bluetooth presences
[11]. These visualisations provide people with information about current or recent
Bluetooth presences.
Radio Frequency Identification (RFID) tags can also be used for presence
identification. In the IntelliBadge project [4], users participating in a conference were
given RFID augmented badges that were used to track them through the conference
rooms. A display at the conference cycled through several visualizations of the resulting
data. RFID tags have the advantage that they are small and can be incorporated into
many existing artifacts. In situations such as a conference, as in the IntelliBadge system,
where people are already obliged to wear a badge, this may be a good choice. Bluetooth,
on the other hand, is a very widely deployed technology and many mobile-phones are
already Bluetooth enabled. This means that it is possible to use the Bluetooth discovery
features to detect presence without requiring the user to carry any additional object (as
with the RFID tags), as most people already carry a mobile phone regularly. Also,
Bluetooth allows the user to manage his presence by turning it on or off at will.
2.2 Presence Self-exposure
Self-exposure corresponds to the ability of the display to obtain information about the
interests, preferences or activities of nearby people. This type of knowledge about the
people that use a place may enable the display to adapt itself to their expectations and
preferences. For this to happen, users must be willing to let the display system know
something about them. This personal information can take many forms: it may be a
reference to a user’s personal webpage, a set of user associated tags, the username for
some social sharing website, a set of interest categories or even personal information,
such as age and gender.
The most common approach for supporting presence self-exposure combines
presence identification with the a priori definition of a user profile that becomes
associated with the identity. This approach was used in the Proactive Displays [13],
were users attending a conference registered their affiliation, interests and personal
webpage before the conference day and were given RFID augmented conference
badges at the conference site. In this system, the user does not have easy access to
their information in order to update it which means that they have less control over
what information the display system uses in a given moment.
Another way to achieve this self-exposure is to use an information device (e.g.
mobile phone) with a custom application that allows users to register a profile. This
application can connect automatically, or on demand, to the display system and
communicate users’ preferences. One example of this is Camera-Phone [22], where a
custom mobile application is used to interact with public displays. This application
may be configured with personal information that is made automatically available to
the display system when a user interacts with the display. One advantage of this
approach is that the information is always available to be updated by its owner.

122 J.C.S. Cardoso and R. José

Bluetooth naming, as described in [8], is yet another alternative for managing self-
exposure. Naming is explored to allow users to enter predefined commands in their
mobile phone Bluetooth name. Since these names can be read by any other Bluetooth
device, this can be used to provide an opportunistic interaction mechanism to any user
since there is no need to install an application. This approach, however, is less suited
for private information since anybody can read the Bluetooth name.
Personal information can also be sent explicitly by the user, using OBEX over
Bluetooth, for example, to push a vCard or other structured text file to the display.
2.3 Suggest Content
The display may offer user the possibility to upload content or references to content
for presentations. By suggesting content, users are implicitly saying that such content
belongs to that place. This is thus a way for the display system to sense the kind of
adequate content for a place. Content may be specified directly or indirectly by the
user: by sending the content itself, e.g., a picture, video, text or audio; or by providing
a reference to the content (e.g. an URL); whatever the means used to suggest content,
the display system will receive or access the content itself and possible meta-data
associated with it.
Many display system provide various alternatives for users to send content in order
to facilitate content submission. WebWall [6], for example, allowed users to suggest
content using SMS, email or a web interface. Plasma Poster [2] is another example of
a display system that allows content (photos, text, web pages) submission through two
interfaces: email and web form. Web Glance [17], a group web browsing system, also
allows several input interfaces to be used: email and instant messaging.
Bluetooth can be used in two ways to send content to a display system: using the
standard OBEX protocol or a custom mobile application. Both Hermes [3] and Snap
and Grab [15] use the OBEX feature to enable users to send pictures (in the case of
Hermes) or any other media type to a display. In both cases, the user just selects the
content on his mobile phone, selects the “send via Bluetooth” command and selects a
particularly named device.
Bluetooth can also be used by mobile applications to communicate with a display
system. The advantage over using just OBEX to transfer files is that a custom
application can be built to interact specifically with a given display thus allowing a
more rich interaction. OBEX has an obvious advantage over a custom application: it
does not need the user to install any additional software on his mobile device and so
allow a slightly more opportunistic interaction.
Content suggestion can be used by display systems in many ways, depending on
the type of content. However, in most cases the display system will be able to
associate, at least, keywords with the content the user submitted (either by gathering
them from the content itself or from meta-data).
2.4 Actionables
Actionables detection corresponds to the ability of the display to detect the user
reactions to any suggested action. A considerable part of the information shown on
public displays is intended to cause people to act [14]. In many cases, the action is
completely unrelated with the interaction supported by the display, and there is no

A Framework for Context-Aware Adaptation in Public Displays 123

way to perceive the efficiency of actionables. However, it is also possible to conceive
actionables that are intrinsically linked to the interaction modes supported by the
public display, and thus obtain feedback on how they are used. This enables the
system to infer interest on the content or services that are being offered.
The concept of actionable is very broad and can take many and very diverse forms.
We will explore in more detail the following approaches: content download, content
control, rating, voting and classification.
Content Download
Content download is a way to get a personal record of something that is currently
displayed. A user may wish to download an item for various reasons: to keep a
permanent record of an item or as a way to inspect an item in more detail if the
display only shows a summary, for example.
Content can be downloaded to the user’s mobile device if a custom mobile
application is provided by the display system that allows browsing and selecting
content to download as in Content Cascade [19] or a user can browse for his
Bluetooth device to send the selected content item in a touch-screen as in the Hermes
Photo Display [3]. A different approach is taken by the Snap and Grab technique [15]
where a user can select an item on the public display by taking a picture of it with a
camera phone and then send it via Bluetooth (OBEX) to the display. The display
system then searches the picture for embedded visual tags that identify the item; if a
tag if found, the associated content is sent back (also via Bluetooth) to the users’
mobile phone.
By downloading an item the user is implicitly saying that he finds that particular
item of some interest, or at least of potential interest.
Content Control
Content control gives users some type of control over the information being
displayed. In a touch-sensitive screen this may result in something very similar to a
browsing experience, where the user can navigate through content and fully control
the display. Other interaction techniques may offer lighter forms of control such as
selecting which video should be presented next from a set of possible alternatives.
Other alternatives may include asking for more details or extending the presentation
time of an item being displayed, or asking for an item currently not being displayed. If
the user asks to skip the current item, the display system can infer that the user does
not find that item particularly interesting and instead wants to see what is next. If the
display shows a list of scheduled items to appear next and the user is able to skip to a
particular item the display system can infer interest on that particular item. Content
control can be achieved by any selection mechanism. A touch screen is an obvious
choice for this. Both Hermes Photo Display and Plasma Poster [2] use a touch screen
interface to let users navigate their content. Jukola [16] also uses a touch screen, but
in this case content is indirectly controlled through voting: users of a bar have the
possiblity to vote on the next music to be played by selecting a music, among a list.
Other selection mechanisms such as the one used by Snap and Grab or Content
Cascade could be used for this purpose. Text commands sent by SMS, email or IM, as
in the In WebGlance [17] system where users send an email IM message to the
display system with a number corresponding to an option on the could also be used.

124 J.C.S. Cardoso and R. José

Content control is also a way to collect users’ implicit interest on an item, similarly
to what happens with content download.
Rating
By rating an item, the user is explicitly saying the he likes or dislikes that item,
depending on the value of the rating. This is a way for the display system to allow a
user to explicitly indicate his preferences. Rating is found on many websites such as
Youtube, Lastfm, Amazon, etc. On public displays, rating can be implemented using
any selection mechanism or through text commands.
Voting
Displays can also collect users’ preferences by crafting polls which allow it to extract
information directly or indirectly from an individual. For example, sports preferences
of a user can be estimated by asking him to vote on his preferred athlete from a list of
athletes from different sports.
As with rating, voting can be accomplished through many different interaction
mechanisms. As an example, Bluevote [1] uses images push via Bluetooth. In this
case the selection command is a picture sent previously by the display system (by
pushing the images to all discoverable Bluetooth devices). Users send back to the
system the picture that corresponds to their vote. Bluevote was used in a conference
setting to allow participants to vote on the best paper award.
Classification
Classification is of a different nature than the previous categories because the result is
not a preference but the association of a description or keywords, for example, with a
given content item. This can be a less natural action for a user, especially for public
items, but it can be provided by displays in a more ludic perspective following the
approach of Games With a Purpose [24]. Classification requires that the user is able to
send free text to the display system and so requires a text submission mechanism such
as SMS, email, IM, Bluetooth names, etc.
2.5 Footprints for Socially-Aware Interactive Displays
The previous sections have highlighted the types of interaction mechanisms that we
may need if we want to gather a particular type of footprint. This section will now
analyse how those multiple footprints can be used to support various types of context-
aware adaptation processes.
Table 1 presents a mapping between the digital footprints and the most widely used
interaction or presence mechanisms that generate those footprints. This mapping can
be used by situated display designers to help choose the interaction mechanisms
provided by the display in order to be able to collect a given set of footprints.
Overall, the entire set of digital footprints constitutes a collection of data which
can be used to characterise the place profile, enabling the display system to adapt its
behaviour to that particular social setting. Regardless of their generic contribution to
this broad adaptation, specific types of footprint can support specific types of adaptive
behaviour. Figure 1 summarizes the relationships that can be established between the
different footprints and possible adaptation processes.

A Framework for Context-Aware Adaptation in Public Displays 125

Table 1. Mapping between interaction mechanisms and digital footprints
Footprint Interaction Mechanism
Presence Detection . Movement detection (proximity sensor; computer-vision)
Presence Characterisation . Face detection with age or gender classification, people counters
Presence Identification . Bluetooth
. RFID
Presence Self-exposure . Bluetooth (profile on device name; a priori profile definition)
. RFID (a priori profile definition)
Suggest Content . Email/IM
. SMS/MMS
. Bluetooth (OBEX; BT Name)
Actionables



. Touch screen (Standard GUI controls)
. Email/IM (Text commands)
. SMS/MMS (Text commands)
. Bluetooth (Text commands, e.g. BT naming; Standard GUI mobile
application)
. RFID (Proximity activation, e.g. Touch & Interact)


Fig. 1. Possible adaptive processes associated with different digital footprints
3 Conclusions
Situated displays cannot rely solely on a static pre-characterisation of the place they
were designed to. They must adapt themselves to their changing environment by
collecting digital footprints that will help in characterising the social context in which
the display is embedded.

126 J.C.S. Cardoso and R. José

In order to be efficient, digital displays need to target their audience’s needs,
expectations and tastes. By collecting digital footprints of people’s interactions,
displays can take a step in this direction.
We have presented an interaction design space that defines a mapping between
interaction mechanisms and their contribution to the generation of digital footprints
with relevance for the characterisation of a place. Each footprint may be used in
isolation or in conjunction with other footprints by digital displays to target specific
aspects of their audience.
Acknowledgements
This work has been supported by “Fundação para a Ciência e Tecnologia” and
“Programa Operacional Ciência e Inovação 2010” (POCI 2010), co-funded by the
Portuguese Government and European Union by FEDER Program and by “Fundação
para a Ciência e Tecnologia” training grant SFRH/BD/47354/2008.
References
1. Bortenschlager, M., Rehrl, K.: BlueVote - A Ubiquitous Audience Voting Service. In:
Adjunct Proc. of the 9th Intl Conference on Ubiquitous Computing UbiComp 2007 (2007)
2. Churchill, E.F., Nelson, L., Denoue, L., Helfman, J., Murphy, P.: Sharing multimedia
content with interactive public displays: a case study. In: DIS 2004: Proc. of the 5th
conference on Designing interactive systems, pp. 7–16. ACM, New York (2004)
3. Cheverst, K., Dix, A., Fitton, D., Kray, C., Rouncefield, M., Sas, C., Saslis-Lagoudakis,
G., Sheridan, J.G.: Exploring bluetooth based mobile phone interaction with the hermes
photo display. In: MobileHCI 2005: Proceedings of the 7th international conference on
Human computer interaction with mobile devices & services, pp. 47–54. ACM, New York
(2005)
4. Cox, D., Kindratenko, V., Pointer, D.: IntelliBadge: Towards Providing Location-Aware
Value-Added Services at Academic Conferences, pp. 264–280 (2003)
5. Electronics, A.: Pressure Mats (2009), http://www.arun-electronics.co.uk/
pressure_mat.htm (visited April 2009)
6. Ferscha, A., Kathan, G., Vogl, S.: WebWall - An Architecture for Public Display WWW
Services. In: The Eleventh International World Wide Web Conference (2002)
7. Jancke, G., Venolia, G.D., Grudin, J., Cadiz, J.J., Gupta, A.: Linking public spaces:
technical and social issues. In: CHI 2001: Proceedings of the SIGCHI conference on
Human factors in computing systems, pp. 530–537. ACM, New York (2001)
8. Jose, R., Otero, N., Izadi, S., Harper, R.: Instant Places: Using Bluetooth for Situated
Interaction in Public Displays. IEEE Pervasive Computing 7(4), 52–57 (2008)
9. Ju, W., Lee, B.A., Klemmer, S.R.: Range: exploring implicit interaction through electronic
whiteboard design. In: CSCW 2008: Proceedings of the ACM 2008 conference on
Computer supported cooperative work, pp. 17–26. ACM, New York (2008)
10. Kwon, Y.H., Vitoria Lobo, N.d.: Age classification from facial images. Comput. Vis.
Image Underst. 74(1), 1–21 (1999)
11. Kostakos, V., O’Neill, E.: Capturing and visualising Bluetooth encounters. In: adjunct
Proceedings of the conference on Human factors in computing systems, CHI 2008 (2008)

A Framework for Context-Aware Adaptation in Public Displays 127

12. Kostakos, V., O’Neill, E.: Cityware: Urban Computing to Bridge Online and Real-world
Social Networks. In: Foth, M. (ed.) Handbook of Research on Urban Informatics: The
Practice and Promise of the Real-Time City. Inf. Science Ref., pp. 195–204. IGI Global
(2008)
13. McDonald, D.W., McCarthy, J.F., Soroczak, S., Nguyen, D.H., Rashid, A.M.: Proactive
displays: Supporting awareness in fluid social environments. ACM Trans. Comput.-Hum.
Interact. 14(4), 1–31 (2008)
14. Müller, J., Krüger, A., Kuflik, T.: Maximizing the Utility of Situated Public Displays. In:
Conati, C., McCoy, K., Paliouras, G. (eds.) UM 2007. LNCS (LNAI), vol. 4511, pp. 395–
399. Springer, Heidelberg (2007)
15. Maunder, A., Marsden, G., Harper, R.: Creating and sharing multi-media packages using
large situated public displays and mobile phones. In: MobileHCI 2007: Proc. of the 9th
Intl. Conf. on Human computer interaction with mobile devices and services, pp. 222–225.
ACM, New York (2007)
16. O’Hara, K., Lipson, M., Jansen, M., Unger, A., Jeffries, H., Macer, P.: Jukola: democratic
music choice in a public space. In: DIS 2004: Proceedings of the 5th conference on
Designing interactive systems, pp. 145–154. ACM, New York (2004)
17. Paek, T., Agrawala, M., Basu, S., Drucker, S., Kristjansson, T., Logan, R., Toyama, K.,
Wilson, A.: Toward universal mobile interaction for shared displays. In: CSCW 2004:
Proc. of the 2004 ACM Conf. on Computer supported cooperative work, pp. 266–269.
ACM, New York (2004)
18. Quividi: Quividi - Automated Audience Measurement of Billboards and Out of Home
Digital Media (2009), http://www.quividi.com/ (visited April 2009)
19. Raj, H., Gossweiler, R., Milojicic, D.: ContentCascade incremental content exchange
between public displays and personal devices. In: The First Annual Intl. Conf. on Mobile
and Ubiquitous Systems: Networking and Services, pp. 374–381 (2004)
20. Sharifi, M., Payne, T., David, E.: Public Display Advertising Based on Bluetooth Device
Presence. In: Mobile Interaction with the Real World (MIRW) in conjunction with the 8th
Intl Conf. on Human-Comp. Interaction with Mobile Devices and Services (2006)
21. Sawhney, N., Wheeler, S., Schmandt, C.: Aware Community Portals: Shared Information
Appliances for Transitional Spaces. Personal Ubiquitous Comp. 5(1), 66–70 (2001)
22. Toye, E., Madhavapeddy, A., Sharp, R., Scott, D., Blackwell, A., Upton, E.: Using
camera-phones to interact with context-aware mobile services. Technical report,
University of Cambridge, Computer Laboratory (2004)
23. Verschae, R., Ruiz-del-Solar, J., Correa, M.: A unified learning framework for object
detection and classification using nested cascades of boosted classifiers. Mach. Vision
Appl. 19(2), 85–103 (2008)
24. von Ahn, L., Dabbish, L.: Designing games with a purpose. Comm. ACM 51, 58–67
(2008)
25. Wikipedia, People counter - Wikipedia, The Free Encyclopedia (accessed 6-April 2009)