An Interactive Visual Query Environment
for Exploring Data
Mark Derthick, John Kolojejchick, Steven F. Roth

Carnegie Mellon University
Robotics Institute, School of Computer Science
5000 Forbes Avenue, Pittsburgh, PA 15213, USA
E-mail: {mad+, jake+, roth+}@cs.cmu.edu
ABSTRACT
Direct manipulation of visualizations is a powerful
technique for performing exploratory data operations
such as navigation, aggregation, and filtering. Its
immediacy facilitates rapid, incremental, and reversible
forays into the data. However it does not provide for
reuse or modification of exploration sessions. This paper
describes a visual query language, VQE, that adds these
capabilities to a direct manipulation exploration
environment called Visage. Queries and visualizations
are dynamically linked: operations on either one
immediately update the other, in contrast to the
feedforward sequence of database query followed by
visualization of results common in traditional systems.
These features are supported by the architectural concept
of threads, which represent a sequence of navigation
steps on particular objects. Because they are tied to
particular data objects, they can be directly manipulated.
Because they represent operations, they can be
generalized into queries. We expect this technique to
apply to direct manipulation interfaces to any object-
oriented system that represents both objects and the
relationships among them.
NOTE: Color versions of the figures are at, e.g.,
http://www.cs.cmu.edu/~sage/UIST97/figure1.gif
1. THE MULTIPLE OBJECT PROBLEM
We are concerned with data visualization systems that
produce “business graphics” consisting of charts, maps,
network diagrams, as well as more sophisticated and
special purpose graphics showing abstract data. It is
often natural to provide the user with an object-oriented
data model to accompany these visualizations. For
instance in .Figure 1, each point on the map represents a
Proceedings of the ACM Symposium on User Interface Software and
Technology (UIST '97), ACM Press, October 1997, to appear.
Copyright 1997 ACM.
single house. Each house has attributes like
neighborhood, latitude, and number_of_rooms.
Many data exploration tasks involve the relationship
among multiple objects. For instance, the database
schema shown in Figure 2(top) has separate objects for
houses, schools, companies, people, and sales. The
schema also includes the attributes for each object type,
and the relationships among objects. For example,
companies have a name, total_sales, and an
incorporation_date. They can be the builders of houses.
Persons can fill any of four roles in a sale: seller_agent,
buyer_agent, buyer, or seller.
Figure 1: All houses, color coded by neighborhood
Proceedings of the ACM Symposium on User Interface Software and Technology (UIST '97), ACM Press, October 1997, pp 189-198.

Databases containing multiple objects like these increase
the complexity of information exploration processes. In
particular, there are three areas of complexity that we
have identified and attempted to address in the
Visage/VQE environment.
1. Browsing to locate information. First, it is difficult
for people to locate information when it is distributed
across multiple objects. For example, locating the
current owners of a set of houses requires knowing
that there is no simple attribute of houses that tells
this. Rather, one must find the most recent sale object
associated with each house and then retrieve the
buyer related to the sale. In other words, users must
be aware of the database schema in order to use the
correct terms for objects, relationships, and attributes
to find relevant information. We’ve provided help for
this problem in VQE with a graphical representation
of the schema that can be directed manpulated (as
illustrated in Figure 2, top).
2. Visualization. Creating visualizations that present the
relationships among attributes from multiple objects
is also complex. For example, Figure 2 (bottom) is a
chart of the lot_size of houses versus the selling_price
of the corresponding sales of those houses. A single
point encodes attributes of two objects: a house and
sale. No previous interactive visualization system
directly supports the construction of visualizations
that are derived from multiple objects. Typically, a
relational database query must be constructed using a
separate interface that joins multiple tables into one
containing attributes from both houses and sales. The
selling_price/lot_size chart can then be generated by
mapping graphical points to records in this new
table. Once the new table is constructed, it is not
obvious how to coordinate visualizations of it with
those of related tables from which it was derived (e.g.
to enable coordinated brushing of points representing
houses on a map with points on a chart representing
the join of houses and sales as in Figures 1 and 2). We
discuss this problem in detail in Section 1.1.
3. Expressing Queries. These databases require users to
retrieve information based on the attributes of
multiple objects and/or the relationships among
objects. For example, one might need to find the
sales of a given house, or to find all the houses with
big lots that sold for a low price. Furthermore, it is
useful for queries to be either extensional or
intentional, to provide different means by which
users can express their requests for objects to be
retrieved.
a) Extensional Expressions. Often, it is easier to
indicate sets of objects via direct manipulation
than to create an expression defining the set. The
bounding box in Figure 1 selects some houses near
the border between Squirrel Hill and Point Breeze.
This is an extensional operation because the set of
points is selected without conveying the
underlying intent of the set. Consequently, if the
border were later changed and the neighborhood
attribute of the affected houses were updated, no
mechanism could infer the change in the selected
set. In general, extensional queries can’t be
reused on different data.
b) Intentional Expressions. An analogous
intentional query would, for instance, select “all
houses within 100 yards of a house with a
different value for neighborhood attribute.”
Intentional queries have a declarative
representation distinct from what they evaluate to
on the current data, and so can be reused.
Our approach to supporting these processes for multi-
object databases is to combine VQE, our visual query
environment, with Visage, our direct manipulation data
exploration environment. In support of the points above:
• VQE includes a schema browser similar to Figure 2
(top), as well as providing a graphical representation
of the query which more easily allows users to find
the attributes they need.
• Visage has been integrated with the SAGE system
[10,11] for automated design of visualizations that
integrate many attributes. SAGE is a visualization
server to Visage, which renders SAGE graphic

Figure 2: Top: the database schema. Bottom: chart comparing
attributes from two different object types. All <house, sale> pairs
related by the sold_in relationship generate a point on the chart.