Book Title
Book Editors
IOS Press, 2003
1
Sensor Fusion-Based Middleware for
Assisted Living1,2
Lorcan Coyle, Steve Neely, Gaëtan Rey, Graeme Stevenson, Mark Sullivan,
Simon Dobson and Paddy Nixon
Systems Research Group
School of Computer Science & Informatics
UCD Dublin, Ireland
Abstract. Systems for home automation can make a vital contribution to the well-
being of individuals requiring moderate amounts of support for day-to-day liv-
ing. Existing systems suffer both from competing and often closed standards bases
and from a message-based architecture that can complicate the development of
flexible applications requiring information from disparate sources. We describe a
knowledge-based pervasive computing middleware and show how it can be used
to provide semantically rich unification over a range of home- and web-based au-
tomation systems.
Keywords. Middleware, Pervasive Computing, Assisted Living
1. Introduction
Home automation can provide major benefits for elderly and mildly disabled people who
require moderate levels of support with day-to-day activities. Whilst a number of au-
tomation systems are available commercially, their use of competing (and often closed)
standards and protocols can impede the creation of “smart homes” in which large num-
bers of heterogeneous devices communicate freely. Such systems also typically use sim-
ple broadcast or hub-and-spoke communications architectures which lead to excessive
centralisation and can impede the free flow of information around a system, especially
as new sensors are added to existing installations. Moreover, there is often a false di-
chotomy between sensor and other data such as web-based information sources and in-
formation stored digitally on PDAs or cellphones. These asymmetries are unhelpful for
application developers.
A common approach to addressing such issues is through the use of middleware.
Traditional solutions such as CORBA or JINI are generally regarded as being too heavy-
weight for smart environments, where devices’ processing and communications capabil-
ities are limited by cost and power considerations.
However, alternative forms of middleware remain an attractive solution to integrated
home automation. In particular, we believe that it is vital for developers to be able to
1This material is based on works supported by Science Foundation Ireland under Grant No. 04/RP1/I544.
2Mark Sullivan is supported by a grant from the Enterprise Ireland commercialisation fund.

2 L. Coyle et al. / Middleware sensor fusion for assisted living
abstract away from the detailed topology, protocols, data formats and control of sensors,
actuators and other information devices, and instead focus on the fusion of information
from disparate sources. This provides great architectural flexibility and has the potential
to improve responses to the noisy and uncertain information typically encountered in
sensor-rich environments.
In this paper we introduce a sensor-fusion-based middleware for smart homes. The
system – ConStruct – treats all devices as either sensors or actuators, allowing arbitrary
home automation equipment to be controlled. All data are given a uniform representation,
and their level of abstraction is raised through the use of knowledge-based data-fusion
techniques. This aids developers in the process of building applications for the smart
home, providing them with information that is semantically closer to their needs than
the raw data provided by individual sensors. Furthermore ConStruct can collect data
uniformly from other sources of interest, such as the web. This provides developers with
a rich body of information with which to drive the automated home. The system is fully
decentralised and decoupled from individual information sources, allowing flexible and
robust use of an evolving device population.
To demonstrate the efficacy of ConStruct in the smart-home and assisted living do-
mains, we describe a smart home-heating system that is under development. ConStruct
collects the inputs from a diverse set of virtual and physical sensors and fuses them into
a soup of RDF data. Applications query this data directly rather than maintaining point-
to-point connections with sensors. This decoupling of consumers of data from produc-
ers enables application developers to work with a single data format (from a single data
source) rather than requiring them to master a number of proprietary formats. This also
allows us to incorporate inferencing abilities into ConStruct that would not be possible
in a system with fragmented data representation.
Section 2 discusses existing middleware solutions and home automation standards.
Section 3 describes the ConStruct architecture and shows how it provides a semantic hub
for home automation systems. Section 4 demonstrates ConStruct’s application to smart
homes through a case study in controlling home environments using a variety of sensors
based on radically different technologies. We show that the system provides developers
with an integrated and homogeneous view of information that is not possible in more
message-based systems. Section 5 concludes with some observations on middleware for
smart-homes and some possible directions for future work.
2. Home automation
The complexity involved in constructing multi-vendor distributed systems can quickly
become unmanageable without support from a tailored infrastructure. The standard ap-
proach is to build abstraction layers with common services made available to developers.
Traditional middleware platforms provide these facilities; middleware in this context can
be broadly defined as a layer between application and system software. Middleware sys-
tems support developers by automating much of the integration between various prod-
ucts and platforms whilst maintaining the integrity of the overall solution in terms of
robustness and reliability.
Middleware for general distributed systems has evolved around a number of distinct
categories. Object based systems such as Java EJB and CORBA provide a platform on

L. Coyle et al. / Middleware sensor fusion for assisted living 3
which to build loosely-coupled object-based systems, complete with operations for reg-
istering objects, discovering new services, transaction handling, security and facilitat-
ing object message passing. Message-oriented middleware decouples client-server com-
munications using the exchange of small messages. More recently, peer-to-peer (P2P)
systems such as Pastry[10] and Chord [12] have become an area of significant research
interest. By removing the need for infrastructural support, P2P systems can potentially
support wireless (and other) ad hoc networks extremely well while distributing the load
and costs of service provision over the node population – at a cost of more complex re-
source location, and no guarantees that particular services be, or remain, available. How-
ever, the removal of a central point of failure results in P2P systems having excellent
fault tolerance properties that can be exploited.
The building and automation industries have a number of standard systems for use
in the deployment of smart spaces. The LonWorks platform from Echelon Corporation is
one such example. LonWorks was designed to address the issues of installation, perfor-
mance, reliability and maintenance of control applications. It is built on a low-bandwidth
communications protocol, LonTalk, facilitating the networking of devices over twisted-
pair, power cables, fibre optics and radio. LonWorks has an affiliated IP tunneling stan-
dard (EIA-852) that can connect devices deployed on LonWorks-based networks to IP-
aware applications and remote network management tools. This is an important step to-
wards fully opening controls systems to Internet-based applications and services, but
requires careful design, installation and management.
BACnet [4] is another example of a data communication protocol designed to sup-
port development of building automation and control networks. BACnet was developed
in an attempt to create a standardised model for of representing devices and interactions
between them with control applications. More recently the oBIX (Open Building Infor-
mation Xchange) standard [6] has emerged: oBIX is an effort aiming to create standard
XML and Web Services to facilitate the exchange of information between intelligent
buildings and applications. A technical committee is working towards defining a standard
web services protocol for exchange of information with the mechanical and electrical
systems in commercial buildings.
Despite these efforts, there are an increasing number of sensor systems and modules
becoming available that adhere to different (or indeed no) standards but which provide
essential information for many applications. Some sensor vendors have focused on IP-
enabled platforms using WiFi or Bluetooth for communications. Other systems provide
proprietary, often research-based interfaces: Smart-ITs [2], Wavenis [1] and i-Bean [9]
provide decentralised networks with simple data flows.
The technological landscape may be summarised as follows. Middleware systems
assume that component nodes have significant processing and communications capabil-
ities – assumptions that are typically not respected in pervasive computing systems, and
perhaps especially in systems intended for use in existing buildings such as are typically
encountered in home automation. Sensor systems often have only simple control and
data interfaces, and do not provide an attractive programming platform for complex ap-
plications. It is hard to build decentralised applications and hard to handle the inevitable
addition of (or failure of) devices in a way that does not require significant application
customisation.

4 L. Coyle et al. / Middleware sensor fusion for assisted living
Moreover the need for self-management, self-description, self-configuration, self-
optimisation and other so-called “self-*” properties points towards the need for more
autonomic and decentralised approaches to middleware targeting pervasive systems [7].
3. ConStruct
We have designed the ConStruct platform as a system for integrating noisy data sources
in clean, dynamic, flexible and semantically well-founded manner. ConStruct provides
applications with a uniform view of information regardless of how that information is
derived, and supports extensive inferencing and sensor fusion within the platform to be
shared between applications.
ConStruct has a fully decentralised architecture; devices within a smart space each
run an instance. Each instance manages the local data provided by sensors (physical
or logical) connected to that device – a “local star” topology in which dumb sensors
are connected to more computationally capable hubs which then exchange information
between themselves. Collectively, the devices maintain a global model of the data within
a smart space. All data are modelled using the Resource Description Framework (RDF)
[15], which provides a standardised way in which to model contextual information and
properties. ConStruct stores and manipulates this data using the Jena Framework [8].
Data exchange between instances of ConStruct is performed using gossiping [13], which
provides high robustness and fine control over network utilisation.
Applications connect to a running instance of ConStruct, and use its services to
obtain required information. Components with domain-specific knowledge may request
and aggregate data from multiple sources in order to contribute new, or refined informa-
tion. All data stored within the system are associated with metadata, which describe data
lifespan and security restrictions.
ConStruct provides a core set of components that provide population management,
data management, querying, and data propagation functionality [11]. Each deployment
contains a component that is responsible for maintaining the local data model, and for re-
moving old data as they expire. A querying service allows applications to request the data
they require to adapt their behaviour. Networking components select peers with which to
share data, whilst components responsible for summarising information restrict the view
of local data to be gossiped and integrated with remote deployments of ConStruct.
3.1. Mapping sensor data
ConStruct views all information sources as sensors that generate RDF triples. This uni-
formity means that to support a particular middleware standard or sensor protocol, a de-
veloper must define a mapping from that system’s internal representation to RDF. (This
has the incidental advantage of making the underlying system’s information more easily
accessible to open services and applications that can interpret RDF or XML information.)
The oBIX standard provides a way to model sensory devices in XML. We use the
example of a simple thermostat from the oBIX specification document [6] to show how
the XML information from oBIX is transformed via XSLT to RDF. This process is il-
lustrated in Figure 1. The oBIX XML snippet models the current state of the thermostat;
its temperature; the target temperature; and the actuator that controls the furnace. An

L. Coyle et al. / Middleware sensor fusion for assisted living 5
XML transform is used to convert this XML to RDF. These RDF values are poured into
ConStruct’s soup of data.
Figure 1. Diagrammatic representation of the transformation from oBIX to RDF.
3.2. Virtual sensors
Although home automation systems focus “inwards” on information sensed within the
home, a properly pervasive approach to assisted living should also look “outwards” to
information available on the web and through other digital channels. In ConStruct these
information sources are simply additional sensors — with the advantage that the emerg-
ing consensus on XML and web services makes much of the information available in
formats that are close to (if not already) RDF.
We believe that this integration of traditional home automation with open access to
web information is vitally important for assisted living applications. One may easily en-
visage scenarios in which an individual’s medical or other requirements are downloaded
securely from remote servers, and summaries of information derived from home-based
sensors are uploaded to provide medical telemetry. ConStruct provides these capabilities
uniformly.
3.3. Programming
In addition to our work on web-sensors we are developing a number of virtual sensors.
Rather than using physical or web data, virtual sensors operate on ConStruct’s soup of
data, using RDQL [14] to query for data of interest. When such data is available, it is
processed by the sensors and the resultant values is added to the soup. One possible
use for this technique is data-aggregation. For example, a virtual sensor can be used
to compute an average temperature from the forecasts of several weather web sensors.
Applications query data from ConStruct in the same way as virtual sensors.
We take the development of virtual sensors a step further by developing an inference
framework that integrates with ConStruct’s soup of data. Such a framework makes it
possible for the system to discover regular patterns in behaviour and to use these to
determine the context that underlies a sequence of actions. An illustrative example of this

6 L. Coyle et al. / Middleware sensor fusion for assisted living
in the context of assisted living would be if an alarm sensor were put in place to ensure
that infirm users made safe night-time bathroom trips. If a person takes an inordinate
amount of time to return from a bathroom trip it may be necessary to trigger an alarm.
4. Case study: heterogeneous integration in a home environment
In this section we explore a simple assisted-living scenario in order to demonstrate how a
more semantics-driven approach to integration simplifies application development. Our
aim is to show how the use of context fusion and symmetric information representation
can provide a significantly enriched environment for pervasive applications over and
above traditional sensor-driven systems.
4.1. The scenario
Consider a smart home that has an autonomous heating control system and a personalised
alarm system. In winter, especially if the house is left unoccupied for any length of time,
steps need to be taken to ensure that the temperature does not fall below freezing (or
else the water pipes may burst). Normally, this would be achieved using thermostat-
controlled central heating. However, this home uses storage heating. Storage heaters use
cheap electricity (usually at night-time) to store heat in ceramic bricks and release it
during the day when electricity is expensive. It is more economical than a typical heating
system if it stores enough heat each night to heat the house during the day.
The typical solution is for a user to manually turn the heater up in cold weather and
down in warmer weather. Our proposal is to use web-published weather data to regulate
the thermostat in the storage heater. We use a web-sensor to retrieve the weather forecasts
and use these predictions to adjust the thermostat to reflect the expected temperature
and store an appropriate amount of heat. This makes the regulation of the thermostat
autonomous. If the weather forecast was incorrect, the thermostat will ensure that the
temperature does not fall below freezing by turning the heaters on whenever necessary,
which will of course necessitate the use of more expensive electricity.
This solution is enough to ensure that the house temperature does not drop too low.
But what if the owner wishes to leave the house for a long period of time or invite a
guest to stay in their absence? We have developed web-sensors that extract data from
published calendar files. By incorporating information about the expected occupancy of
the house and altering the thermostat temperature accordingly we improve the efficiency
of the heating system.
4.2. Basic solution
The use of the RDF standard for modelling context provides a common level of abstrac-
tion with which to represent information generated within a smart-space. This allows all
data to be managed, manipulated, and disseminated independently of the technologies
used to acquire it. This allows us to deal with the heterogeneity of sensors and devices to
be found within a smart-environment.
ConStruct is a fully decentralised infrastructure, with participating device clusters
each running an instance of the ConStruct software. The use of decentralised algorithms

L. Coyle et al. / Middleware sensor fusion for assisted living 7
for population management and data propagation allows a smart-space utilising Con-
Struct to evolve in size gracefully, without the need for centralised administration.
The arrangement of devices in a smart space may change frequently as devices turn
on and off, users change location, and new data sources are integrated. By using estab-
lished discovery protocols (such as Zeroconf [3]) to detect running instances of Con-
Struct, and a decentralised protocol for data propagation, ConStruct is robust to such
changes. As the infrastructure is fully decentralised, the failure or removal of one deploy-
ment has minimal impact on the smart-space. New devices need only detect a running
instance of ConStruct to join the smart-space.
These features provide a solution to the problems of sensor heterogeneity, device
volatility, data propagation, and application integration within smart-environments. The
combination of these features make ConStruct a robust infrastructure, capable of pro-
cessing and distributing data from a potentially large numbers of sensors.
5. Conclusions and ongoing work
Existing home automation systems exhibit a tight coupling between applications and the
sensor networks that they utilise. As it is desirable to support both the integration of such
systems and the sharing of sensed data across multiple applications, there is a clear need
for middleware level support.
We introduce a sensor-fusion based middleware for smart-homes called ConStruct.
ConStruct treats all components of a smart environment as sensors or actuators and maps
data from these components into a unified format. This provides us with a common level
of abstraction which simplifies the process of manipulating and querying for data.
ConStruct extends the traditional model of home automation systems, which focus
on information sensed within the home to include information sensed from external digi-
tal media. As increasing amounts of information is published in digital formats and made
available, additional value-added features can easily be integrated with assisted-living
applications.
We describe an assisted-living scenario that demonstrates how ConStruct fuses
sensed data and facilitates its consumption. Our next step is to continue building up a
repository of physical and virtual sensors for use in the smart-home environment. We
are complementing this work by the development of an inference framework that per-
forms aggregation of data and deals with inconsistencies that may arise when the same
type of data is produced from different sensors (e.g. location sensors providing data that
says a person is in two places at once) [5]. We are also developing a set of heuristics for
evaluating the performance of ConStruct.
Acknowledgments
This work was partially supported by the grants “Secure and Predictable Pervasive Com-
puting” from Science Foundation Ireland and “A Platform for User-Centred Evaluation
of Context-Aware Adaptive Services” from Enterprise Ireland.

8 L. Coyle et al. / Middleware sensor fusion for assisted living
References
[1] Coronis systems, wavenis technology.
http://www.coronis-systems.com/descriptif.php?id=descr_tech.
[2] The Smart-Its project. http://www.smart-its.org/.
[3] Zeroconf. http://www.zeroconf.org/.
[4] S. Bushby and H. Newman. The bacnet communication protocol for building automation
systems. ASHRAE Journal, 33(4):14–21, April 1991.
[5] S. Dobson, L. Coyle, and P. Nixon. Hybridising events and knowledge as a basis for building
autonomic systems. In Journal of Trusted and Autonomic Computing. September 2006, 2006.
To appear.
[6] B. Frank. obix specification. Working Draft 0.8, December 2005.
[7] J. Kephart and D. Chess. The vision of autonomic computing. IEEE Computer, 36(1):41–52,
January 2003.
[8] B. McBride. Jena: Implementing the RDF model and syntax specification. In Proceedings of
the 2nd International Workshop on the Semantic Web, Hong Kong, may 2001.
[9] S. Rhee, D. Seetharam, S. Liu, N. Wang, and J. Xiao. i-beans: An ultra-low power wireless
sensor network. In Interactive Poster in the Fifth International Conference on Ubiquitous
Computing (UBICOMP), 2003.
[10] A. Rowstron and P. Druschel. Pastry: Scalable, Decentralized Object Location, and Routing
for Large-Scale Peer-to-Peer Systems. Lecture Notes in Computer Science, 2218:329–350,
2001.
[11] G. Stevenson, L. Coyle, S. Neely, S. Dobson, and P. Nixon. Construct — a decentralised
context infrastructure for ubiquitous computing environments. In IT&T Annual Conference,
Cork Institute of Technology, Ireland, 2005.
[12] I. Stoica, R. Morris, D. Karger, F. Kaashoek, and H. Balakrishnan. Chord: A Scalable Peer-
To-Peer Lookup Service for Internet Applications. In Proceedings of the 2001 ACM SIG-
COMM Conference, pages 149–160, 2001.
[13] S. Voulgaris, M. Jelasity, and M. van Steen. A Robust and Scalable Peer-to-Peer Gossiping
Protocol. In The Second International Workshop on Agents and Peer-to-Peer Computing
(AP2PC), 2003.
[14] W3C. RDQL - A Query Language for RDF. http://www.w3.org/Submission/RDQL/.
[15] W3C. Resource Description Framework (RDF). http://www.w3.org/RDF/.