Learning Physically-Instantiated Game Play Through Visual Observation
Andrei Barbu, Siddharth Narayanaswamy, and Jeffrey Mark Siskind
Abstract— We present an integrated vision and robotic sys-
tem that plays, and learns to play, simple physically-instantiated
board games that are variants of TIC TAC TOE and HEXA-
PAWN. We employ novel custom vision and robotic hardware
designed specifically for this learning task. The game rules can
be parametrically specified. Two independent computational
agents alternate playing the two opponents with the shared vi-
sion and robotic hardware, using pre-specified rule sets. A third
independent computational agent, sharing the same hardware,
learns the game rules solely by observing the physical play,
without access to the pre-specified rule set, using inductive logic
programming with minimal background knowledge possessed
by human children. The vision component of our integrated
system reliably detects the position of the board in the image
and reconstructs the game state after every move, from a
single image. The robotic component reliably moves pieces both
between board positions and to and from off-board positions
as needed by an arbitrary parametrically-specified legal-move
generator. Thus the rules of games learned solely by observing
physical play can drive further physical play. We demonstrate
our system learning to play six different games.
I. INTRODUCTION
Children learn to play games by watching others play.
While both formal board games, like CHESS, CHECKERS,
and BACKGAMMON, and less formal play like HOPSCOTCH,
TAG, and DODGEBALL all have well defined rules that
children ultimately come to know, they are rarely told those
rules explicitly. Knowledge of how to play many classic
board games is largely passed down culturally, with chil-
dren never reading, and often even explicitly ignoring, the
formally-specified rules (e.g., Monopoly R
). We are engaged
in a long-term research effort to emulate on robots this
ability to learn to play games by observing others play. The
work presented here is part of a larger effort to ground
learning, reasoning, and language in visual perception and
motor control. Physical instantiation is crucial to our effort of
situating learning, visual perception, and manipulation in the
real world. We want physical robots to play a physical game
where knowledge of game play allows their vision systems to
determine game progress and motor systems to effect game
progress. We also want a physical learner to visually observe
that play to learn the game rules and ultimately be able to
use the learned rules to support physical game play.
Our long-term vision for this overall task is depicted in
Fig. 1. In this task, two robotic agents, the protagonist
and antagonist, play a board game like CHESS. A third
robotic agent, the wannabe, does not know the rules of
the game but must infer the rules by visually observing
The authors are with the School of Electrical and Computer
Engineering, Purdue University, West Lafayette, IN, 47907, USA
fabarbu,snarayan,qobig@purdue.edu
the play of the protagonist and antagonist. The wannabe
must then use these rules for further physically-instantiated
play. In the long term, we wish to be able to do this for
a wide variety of off-the-shelf game hardware for a wide
variety of common physically-instantiated board games. Our
objective is to learn to play legally, not necessarily well.
Expert computer game play is one of the most extensively
studied and successful sub-disciplines of AI. Our goal is
orthogonal to that enterprise.
We have constructed a novel custom robot to support this
enterprise, and have used this robot to successfully learn six
different physically-instantiated games. While one long-term
goal is to learn a wide variety of common board games,
like CHESS, CHECKERS, BACKGAMMON, and GO, with
differing physical game hardware, the work presented in
this paper is limited to games which share the same game
hardware. And while another long-term goal is to use three
separate robotic agents to play the roles of protagonist,
antagonist, and wannabe, the work presented here uses a
single robot to play all three roles. We do however, use a
unique capability of our novel custom robot to simulate play
by multiple distinct agents by robotically moving the camera
to image the game play from different viewpoints.
II. OUR CUSTOM ROBOT
We have designed a custom robot and built three copies
thereof, one of which is shown in Fig. 2. While much
of our robot is constructed with off-the-shelf parts, many
crucial parts were custom designed, milled, or repurposed to
meet the particular needs of the game-playing task. The two
most-novel parts are the overall housing and camera-mount
assembly. The overall housing consists of a two-level wood
platform, where the upper level constitutes the game-play
surface and the lower level serves as the mounting point for
the camera assembly. A 5 DOF arm with two independently-
controllable fingers, is bolted to the upper level. The size of
the overall housing and the arm link lengths were designed
to support game play with off-the-shelf game hardware.
Our robot hand contains a number of sensors to support
fine motor control for manipulating game pieces: a palm-
mounted camera, an ultrasonic range sensor, a laser pointer,
and tactile force sensors on each fingertip. The camera as-
sembly consists of a pair of pan-tilt USB webcams mounted
on a 1 DOF pendulum arm which is in turn mounted on
a servo base bolted to the lower level. We found Logitech
QuickCam Orbit cameras well-suited to our task, as we were
able to strip them down to a lightweight assembly containing
the camera, pan-tilt motors, and electronics, allowing them to
be mounted on the pendulum arm. This allows them to pivot,