A Framework for Empirical Evaluation of Model Comprehensibility


Jorge Aranda, Neil Ernst, Jennifer Horkoff, and Steve Easterbrook
University of Toronto, Canada
{jaranda, nernst, jenhork, sme}@cs.toronto.edu


Abstract

If designers of modelling languages want their
creations to be used in real software projects, the
communication qualities of their languages need to be
evaluated, and their proposals must evolve as a result
of these evaluations. A key quality of communication
artifacts is their comprehensibility. We present a
flexible framework to evaluate the comprehensibility of
model representations that is grounded on the
underlying theory of the language to be evaluated, and
on theoretical frameworks in cognitive science.

1. Introduction

Over the past decades, hundreds of conceptual
modelling languages have been proposed as tools to
understand and communicate software project
information [14]. We have a wealth of notations at our
disposal to represent almost any kind of information
we wish, from machine states to stakeholder goals. Yet
the use of these modelling languages in real software
projects and their adoption rate by the software
industry are still very low [7].
An important cause of this usage problem may be a
lack of attention to the extent to which these languages
enable effective communication among their users.
Models have many uses, but one of the most prominent
is serving as communication artifacts in software
teams. In fact, if they have one purpose, for most
languages, it is communicating ideas.
The effectiveness of software models depends on a
number of communication qualities such as: Cost of
production, comprehensibility, speed of ‘decay’ (loss
of synchrony with the content it represents), and
steepness of their learning curve. If a language is
deficient in several of these qualities, then it does not
matter whether it has a high expressive power or well-
formalized semantics; it will not be used for
communication purposes.
Considering models as communication artifacts
raises an important issue. Even the simplest models of
communication available [11] require a receiver to
decode and assimilate the message for the
communication instance to be successful. A
communication event does not stop with the
transmission of an encoded message. In practical
terms, creating and sending a diagram to somebody
may lead us to believe that we have communicated its
information to that person; but if the diagram is not
read, processed, and assimilated correctly by the
receiver, the communication instance has failed.
For this reason, an essential quality of
communication artifacts is their comprehensibility.
Documents and diagrams that are cryptic, misleading,
or vague will not serve their communication purpose.
Therefore, it is important to bring comprehensibility,
along with other communication qualities, to the
forefront of the modelling language debate.
Unfortunately, as we will discuss in Section 4, there
have been very few careful empirical studies that
evaluate the comprehensibility of software modeling
languages. When it is considered at all, judgments
about model comprehensibility are often very
subjective and have little regard for empirical validity.
In this paper we present an empirical framework to
evaluate model comprehensibility. The framework,
presented as a sequence of steps and guidelines, is
intended to guide evaluators to address the challenges
of studying a construct as subtle and complex as
comprehensibility. We assume that any researchers
who apply it will have some empirical software
engineering expertise, and access to expert modellers
of the language of their choice.

2. The comprehensibility construct

2.1. Challenges to define the construct

The first challenge for evaluators of model
comprehensibility is to define the meaning of the
construct: it is an intuitive concept, but very difficult to
define. The naive view (“Can I make sense of this
document?”) breaks down when we try to