Multi-touch Interface for Controlling
Multiple Mobile Robots

Abstract
We must give some form of a command to robots in
order to have the robots do a complex task. An initial
instruction is required even if they do their tasks
autonomously. We therefore need interfaces for the
operation and teaching of robots. Natural languages,
joysticks, and other pointing devices are currently used
for this purpose. These interfaces, however, have
difficulty in operating multiple robots simultaneously.
We developed a multi-touch interface with a top-down
view from a ceiling camera for controlling multiple
mobile robots. The user specifies a vector field followed
by all robots on the view. This paper describes the user
interface and its implementation, and future work of
the project.
Keywords
multi-touch interface, multiple-robot operation, human
robot interaction, home robot, entertainment robot
ACM Classification Keywords
H5.2. Information interfaces and presentation (e.g.,
HCI): User Interfaces – Interaction styles; I.2.9.
ARTIFICIAL INTELLIGENCE: Robotics – Commercial
robots and application
1. Introduction
All robots, including those that do their tasks
autonomously, do not work without an instruction by
Copyright is held by the author/owner(s).
CHI 2009, April 4–9, 2009, Boston, Massachusetts, USA
ACM 978-1-60558-247-4/09/04.
Jun Kato
The University of Tokyo
School of Science,
Dept. of Information Science
jun.kato@acm.org
Daisuke Sakamoto
The University of Tokyo
Graduate School of Information
Science and Technology
d.sakamoto@gmail.com
Masahiko Inami
Keio University
Graduate School of Media Design
inami@designinterface.jp
Takeo Igarashi
The University of Tokyo
Graduate School of Information
Science and Technology
takeo@acm.org

users. We therefore need interfaces for giving
instructions to them.
Single robots are normally controlled with joysticks,
keyboards, and other pointing devices. However, with
advances in robotics, variations of user interfaces for
these purposes have become wider. For example,
multimodal interfaces such as combination of hand
gestures and speech for one assistant robot [1], and a
portable interface using personal digital assistant (PDA)
for mobile robots [2] have been proposed. This system
allows users to navigate a robot with waypoints
projected on the screen. Recently, user studies of
intuitive interfaces [3] have been performed, in which
motor impaired users have a robot grab things with
laser pointers and a touch-screen with buttons on it.
Among these studies, it has been concluded that the
invention of advanced user interfaces helps operating
single robots.
Handling tasks with multiple robots is also desirable,
because they can do various tasks with greater
efficiency than a single robot. However, multiple robots
substantially increase amount of information exchange
with their users who have to maintain situational
awareness and continue operation. It often makes the
manner of operation complex and difficult. Users have a
limitation in the capability of their attention, so they
cannot see too much information displayed either at
the same time or time-multiplexed. Therefore, many
user interfaces for operating single robots do not work
effectively. Designing interactions between people and
multiple robots to achieve their effective cooperation
has been a difficult research issue.
Upon the cooperation of people and multiple robots,
existing studies have tried to give robots some
intelligence. This approach aims to allow for limited
resources of people to care the situation and make
orders continuously. Some have continued studies
which only tell robots initial state and make them work
autonomously [4]. Others like Fong et al. insisted that
completely autonomous approaches are not yet feasible,
and that robots should engage in dialogue with their
users when required [5]. Their study indicates two
factors for effective cooperation of people and robots.
The first factor is that roles and responsibilities against
tasks are clearly separated between users and robots.
It is generally said that users should have responsibility
for global tactics, and robots for local tasks. When the
distinction between global and local tasks is unclear,
problems occur for both the robot and the user. The
second factor is that users can command robots easily
as possible. Here we need richness of user interfaces
cultivated in the field of HCI. Discussion of Driewer et al.
describes how the user interface in human-robot teams
should be [6]. They pointed out that in teams
consisting of people, robots and their supervisor, the
use of graphical user interfaces (GUI) greatly affect the
performance of their tasks.
In this paper, we propose an intuitive interface using a
multi-touch display to control multiple mobile robots
simultaneously. Users get a top-down view from a
ceiling camera in real time, which is virtually overlaid
with a 2-dimensional vector field. All robots follow the
vector field. Users can manipulate this field by touching
and passing their hands on the display. Therefore, they
can easily control all robots through the multi-touch
display.