OVERCOMING THE KEYHOLE IN
HUMAN-ROBOT COORDINATION:
SIMULATION AND EVALUATION
Martin Voshell, David D. Woods
Institute for Ergonomics, The Ohio State University, Columbus, OH
and
Flip Phillips
Eye, Brain & Vision Laboratory, Skidmore College, Saratoga Springs, NY
When environment access is mediated through robotic sensors, field experience and naturalistic studies show
robot handlers have difficulties comprehending remote environments - they experience what domain practi-
tioners often call a ‘soda straw’. This illustrates the keyhole effect in Human Robot Interaction, a CSE phe-
nomena studied in the context of large virtual data space interfaces and the current research seeks to reduce
this effect. A simulation for human-robot coordinated search and rescue was created based on WTC re-
sponse experiences. Pilot studies showed traditional performance measures to be inadequate in analyzing
control and exploration tasks therefore a novel analysis approach based on fractal path tortuosity was devel-
oped. New interface concepts for helping remote observers perceive environmental affordances were then
tested using the simulation environment and evaluation measures. These studies look to concepts based on
Gibsonian principles to reduce keyhole effects in control interfaces to enhance remote functional presence in
Human-Robot Coordination.
INTRODUCTION
There are many challenges to designing successful
human-robot systems (Woods, Tittle, Feil, Roesler, 2004). When
the robot is a stand in for a remote human observer, the natu-
ral dynamic relationship between properties of the scene being
explored and the human perceptual system is broken. This de-
coupling undermines the remote observer’s perception of af-
fordances in the scene (Gibson, 1986) and is fundamental to
the remote perception problem (Tittle, Roesler, Woods, 2002).
This is illustrated in recent cases of HRI where remote ob-
servers experience various difficulties understanding the envi-
ronment being traversed by a robotic system which severely
limits their effectiveness (Murphy, 2003, 2004).
Practitioners often complain about what they call the
‘soda straw effect' due to the limited angular view available
from the robots’ cameras (Casper and Murphy, 2003). Like
many of the CSE themes we deal with, the “soda straw” serves
as a succinct visual metaphor to a much larger cognitive prob-
lem, the keyhole effect. Keyhole effects are indicators of data
overload and typical consequences include missing new events,
increased difficulty in navigating novel environments, and creat-
ing gaps or incoherent models of explored space (Woods and
Watts, 1997). Keyhole problems arise from the fact that typical
virtual data displays, or in this case remote feeds from a robot,
sever the coordination between the focal attention and the
orienting perceptual functions on the side of the remote par-
ties that help people to fluently know where to look next, de-
spite the potential for novel interesting events to encroach
ongoing activities. Thus, keyhole problems do not simply refer
to the optical field of view, but serve as indications of deeper
seeded problems in control and coordination as well.
When stuck looking through the ‘soda straw' opera-
tors have a difficult time understanding spatial layout. Opera-
tors easily miss alleys, landmarks are tough to discern, it is diffi-
cult to locate human targets for rescue, and the handler is usu-
ally forced to manually switch and integrate multiple views in
the interface. Safe and successful navigation is more than just
looking where one is going: in the natural world humans tend
to sample everywhere around where they are going in a con-
text sensitive way.
For a simple example, contrast how you would direct
your gaze as you turn to climb a flight of stairs versus how a
robotic platform positions their cameras during the same ma-
neuver. Generally, the robot camera either points at each step
one at a time or remains pointed at the ceiling as the robot
climbs, whereas people direct their attention and shift their
gaze in tight coordination with the affordances present in the
environment given their goals and context to the task (e.g.,
when to look for activity at the top of the stairs, seeing and
reacting accordingly to debris and potential obstacles along the
stairs).
The challenge for human-robot control and coordina-
tion is to overcome these keyhole effects and enhance remote
observers’ understanding of the environment being traversed
by the robotic system. This research addresses these remote
vision challenges by creating a simulation of HRI in a search
and rescue situation, developing new measures of HRI per-
formance based on fractal path analysis, and testing new con-
cepts in interface design to overcome keyhole effects and avoid
getting lost in data overload.
From Field work to Simulation
Field experience in search and rescue had shown that
keyhole effects and other problems in remote perception sig-
nificantly reduce the potential benefits of rescue robots, for
example, see Casper and Murphy's experiences with rescue
robots at the World Trade Center (Casper and Murphy, 2003).

A virtual human-operated robot assisted search and
rescue simulation was created based on Murphy’s experiences
from the WTC response and set within a simulated recon-
struction of the NIST (National Institute for Standards and
Technology) Orange reference test facility for autonomous
robots. Michael Lewis began using Unreal Tournament video-
game based simulations for USAR applications citing the bene-
fits of realistic physics, low cost of overhead, and high-fidelity
graphics (Wang, Lewis, and Gennari, 2003). We collaborated
with Lewis and started working with a basic model of the NIST
Orange Arena. We heavily modified the Orange environment
to add many new obstacles, ambiguities, and world geometry.
Within the simulation, operators had to perform multiple ex-
ploratory tasks analogous to actual search and rescue missions
while facing many of the same physical and visual constraints
seen in the field. The C/S/E/L Unreal code allows an investiga-
tor to introduce various robotic platforms, sensors, and to
rapidly develop and test new interfaces and camera arrange-
ments in a variety of dynamic environments. A high-level quan-
titative analysis framework was built into the engine to capture
event information and three dimensional position data for per-
formance evaluation.
Novel Analysis Methods
What became evident from pilot studies was that
traditional measures for analyzing a robot handler’s perform-
ance in a complex environment with multiple tasks were not
very informative. Inter-subject variability in navigation was high
and gathering completion times and items collected did not
capture the intricacies and problems inherent in search and
Figure 1. Paths shown with increasing frac-
tal dimensions.
rescue. To meaningfully evaluate goal-oriented pathfinding ac-
tivity, we developed measures borrowing techniques from two
relatively diverse fields, ecological perception and physiological
entomology. Visually guided translation is well researched in
perceptual psychology and the most valuable data to focus on
are the affordances in a given environment (Gibson, 1986).
Optic flow during translation contributes greatly in most any
visually guided behavior (Warren, Morris, and Kalish, 1988) and
one immediate application of this in robot control is the
handler/robot's ability of perceiving aperture affordances. For
this, we chose to look at path approach velocity transitions in
relation to the ‘pasability' of apertures and obstacles. The
other metric of analysis examines at the fractal dimension of
the handler/robot's path (See Figure 1). Adapting a similar
method to that first used by Dicke and Burrough (Dicke and
Burrough, 1988) to characterize the tortuosity of spider mite
spatial exploration, deviations in the fractal dimension of the
robot/handler's goal–directed path through a complex envi-
ronment relative to various ambiguities and obstacles provides
a metric that captures a very rich set of descriptive behaviors.
Changes in the fractal dimension of the path provide significant
insight into the handler/robot's search efficiency, path tortuos-
ity, and overall space utilization in relation to handler goals and
overall characteristics of the environment.
Proposed Interface: Perspective Folding
One design concept based on field hardware con-
straints we have proposed to attempt to re-couple perception
and action cycles and help overcome the keyhole by making
these affordances more salient is Perspective Folding (see Fig-
ure 2).
By folding the display screen around the remote ob-
server and mapping camera sensor output, we re-embed the
observer in the scene and re-introduce peripheral optic flow
cues that signal movement relative to the environment. explic-
itly in front of the observer. In contrast to a single wide angle
camera or an interface to switch among different camera feeds,
in the Five-Fold version of Perspective Folding an array of five
cameras, each oriented at different angles, provides a wrap
around effect. By doing this, we are attempting to perceptually
integrate multiple views into a global reference frame. The
camera orientations are preserved on the robot handler's dis-
play, presented in depth, and do not just provide a larger field