1
A revised version of this technical report appears as Chapter 3 (pp. 85-127) in K. Marriott & B. Meyer (Eds.),
Visual Language Theory, Springer-Verlag Publishers, Berlin, 1997.
2
Author to whom correspondence should be addressed. Email: narayan@eng.auburn.edu
VISUAL LANGUAGE THEORY: TOWARDS A
HUMAN-COMPUTER INTERACTION
PERSPECTIVE 1
N. Hari Narayanan2
Roland Hübscher
Technical Report CSE97-05
September 30, 1997
Visual Information, Intelligence & Interaction Research Group
Department of Computer Science & Engineering
Auburn University
Alabama 36849-5347
USA
http://www.eng.auburn.edu/department/cse/research/vi3rg/vi3rg.html

1VISUAL LANGUAGE THEORY: TOWARDS A HUMAN-COMPUTER
INTERACTION PERSPECTIVE
N. Hari Narayanan
Roland Hübscher
Abstract
The main reason for using visual languages is that they are often far more convenient to the user
than traditional textual languages. Therefore, visual languages intended for use by both computers
and humans ought to be designed and analyzed not only from the perspective of computational
resource requirements, but equally importantly, also from the perspective of languages that are
cognitively usable and useful. Theoretical and practical research on visual languages need to take
into account the full context of a coupled human-computer system in which the visual language
facilitates interactions between the computational and the cognitive parts. This implies that
theoretical analyses ought to address issues of comprehension, reasoning and interaction in the
cognitive realm as well as issues of visual program parsing, execution and feedback in the
computational realm. The human aspect is crucial to visual languages, and therefore we advocate a
correspondingly broadened scope of inquiry for visual language research. In this chapter we
describe aspects of human use of visual languages that ought to be important considerations in
visual language research and design, and summarize research from related fields such as software
visualization and diagrammatic reasoning that addresses these issues. A framework consistent with
the broadened scope of visual language research is proposed and used to categorize and discuss
several formalizations and implemented systems. In the course of showing how a sample of
current work fits into this framework, open issues and fruitful directions for future research are
also identified.
I. Why Visual Languages?
In the various disciplines of human inquiry, be it science, arts or humanities, the answer to this
question is obvious. Our visual systems have evolved over eons to become fine tuned in being able
to perceive patterns, sometimes even where none exist, and our brains can make sense of them
through recognition and interpretation. Conversely, our brains are adept at converting thoughts into
visual representations, some of which are constrained by conventions, while others are brilliantly
executed depictions governed by nothing other than the imaginations of an artist. In fact, visual
thinking (Arnheim, 1969) is deeply embedded in the psyche of every culture, and visual languages
predate the development of textual ones (Tversky, 1995).
The ease with which we create and comprehend visual representations is surely a significant factor
motivating research in the area of visual languages, and this is articulated in many different ways
(Blackwell, 1996). Most of this research has been on visual programming languages in the past.
However, the adeptness of the human visual system masks the computational complexity of
parsing, interpreting, and operationalizing the semantics carried by visual representations - from
the simplest of diagrams to complex real world scenes. Computers are not yet capable of vision at
the human scale. Neither are these machines capable of generating visual representations that depict
abstract thoughts and ideas except under the direction of a human (McCorduck, 1991). Chapters of
this book demonstrate the difficulty of formalizing visual languages so that their computational