Context for Semantic Metadata
Kenneth Haase
beingmeta, inc. & Media Lab Europe
68 Bailey Street
Boston, MA 02124
kh@beingmeta.com

ABSTRACT
This article argues for the growing importance of quality
metadata and the equation of that quality with precision and
semantic grounding. Such semantic grounding requires
metadata that derives from intentional human intervention as
well as mechanistic measurement of content media. In both
cases, one chief problem in the automatic generation of semantic
metadata is ambiguity leading to the overgeneration of
inaccurate annotations. We look at a particular richly annotated
image collection to show how context dramatically reduces the
problem of ambiguity over this particular corpus. In particular,
we consider both the abstract measurement of “contextual
ambiguity” over the collection and the application of a particular
disambiguation algorithm to synthesized keyword searches
across the selection.
Categories and Subject Descriptors
H.2.1 Logical Design, H.2.3 Data Description languages, H.2.4
Multimedia databases, H.2.7 Database Administration: Data
dictionary/directory, H.3.3 Information Search and Retrieval,
H.3.7 Digital Libraries, I.2.7 Natural Language Processing, K.1
The Computer Industry
General Terms
Management, Measurement, Performance, Economics,
Algorithms.
Keywords
Metadata, information retrieval, context, disambiguation,
multimedia databases.

1. The Value of Metadata
Detailed and precise metadata will be the key to the next
generation of applications that will realize the potential of the
new digital media. Especially with largely opaque media
(image, video, audio), metadata provides the handles by which
programs can search, arrange, and repurpose the digital media
that are quickly becoming ubiquitous. However, the generation
of that metadata and the economics of that production and
application remain problematic.



We believe that the economics of metadata are subject to a
principle analogous to Metcalfe’s law for the economics of
network technologies [5][12], in particular that
“the value of metadata rises as the product of the log of the
corpus size and the log of the size of the user community”
For example, good metadata would not be a crucial issue for
small databases (100s or 1000s of items) or a handful of users,
since a simple organizational scheme (based on time, place,
keywords, etc.) could be combined with personal knowledge to
allow fast, relatively reliable, retrieval and identification of
relevant images. However, as the total number of images grows
or the community size increases, the value of metadata increases
substantially.
Applying this principle to the growing pool of digital media and
clear market trends (omnipresent cameras, huge disks,
ubiquitous broadband), suggests an oncoming transition in the
traditional economics of data and metadata.
2. The Metadata Twist
While the economic significance of metadata increases with the
size of the available content, the average economic value of the
content must, at the same time, decrease with the amount of
available content. This pair of trends leads to a projection that
we call the “Metadata Twist”, illustrated in Figure 1.
As media technologies improve and spread, there will be a
gradual transformation where metadata will become more
valuable (on average) than the content it describes. This
counterintuitive twist arises as the human resource of time and
attention remains fixed while the pool of accessible media
increases at least exponentially.
The potential in this transformation is enormous and motivates
the investment in technologies and capabilities for dealing with
the metadata that will become increasingly valuable.
Considered carefully, it also focuses attention on the importance
of high-quality precise metadata.
3. Quality, Precision, and Semantic
Metadata
Not all metadata are created equal. One useful definition of
metadata is “any data which conveys knowledge about an item
without requiring examination of the item itself.” Because
metadata derives its value from saving human time and
attention, it must be effective at distinguishing relevant and
irrelevant or redundant content. For automatic storytelling,
quality metadata is even more important, since a given

Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
MM’04, October 10–16, 2004, New York, New York, USA.
Copyright 2004 ACM 1-58113-893-8/04/0010...$5.00.
204

presentation may rely on complex sequences of discriminations
regarding features and connections which can serve narrative
goals.
One of the chief characteristics of metadata quality is precision:
metadata needs to be precise enough to effectively distinguish
between different items so that it can select the right one without
requiring that a person examine the items (absorbing their
precious time and attention). Precision is equally important to
determine what items should be excluded as included, whether
for reasons of irrelevance, inappropriateness, or redundancy. In
either case, the more precise the metadata, the more valuable (in
the terms described above) it is.
There are two significant problems with precise metadata. First,
in conventional databases and metadata models, increasing
precision of description reduces the overall recall performance.
If you describe an image as a “German Shepard,” a person
searching for pictures of “Pets” will not find it. Second, precise
metadata may require some human attention to produce it, the
same attention we’re trying desperately to save by using it in the
first place.
The problem of diminished recall (“German Shepards” vs.
“Pets”) can be addressed in large part by using “semantic
metadata” which links related terms to one another, so that
“German Shepard” is connected to “Pets” in some manner.
Semantic metadata differs from traditional taxonomies or
structured thesauri in two important ways:
• it provides articulated patterns of reference, describing
(even if only in natural language) how terms map to
content; and
• it provides operational rules of inference explaining how
and when terms can be expanded to other terms.

These criteria are interdependent. For instance, to insure that we
can expand a term like “Fish” appropriately, we need to
distinguish “Fish, the food” from “Fish, the animal”.
Conversely, because we may distinguish “German Shepard”
from “Labrador Retriever,” we need to be able to infer that they
are both kinds of pets (or at least domesticated animals).
While we have focused here on terminological precision, other
sorts of precision are equally important. Davis [2] makes a
similar argument for stream-based, rather than clip-based, video
representations (in which all annotations have time indices), for
the same reasons of more exact retrieval and better automatic
manipulability.
In conclusion, precise description of large collections requires
semantic metadata and the application of semantic metadata has
its own requirements. We consider these in the next section,
especially the need for multiple schematizations or taxonomies
and the need (into the foreseeable future) for human annotation
of content.
The second issue, the effort required for precise annotation, is
potentially more serious. The argument for precise annotation is
that we are strategically shifting time and attention from the
search process to the initial annotation process. If the number of
searchers is large and their time is valuable, this can be a strong
argument. On the other hand, when the number of documents is
very large, the cost of precisely annotating each one (even ones
which searchers will never see) may outweigh this advantage.
However, it is important to not take naïve assumptions about the
cost of precise annotation for granted. As we will discuss
below, new techniques --- such as the use of context --- can
dramatically reduce the cost of annotation.
4. Metadata and Purposes
Metadata, and especially semantic metadata, is necessarily a
representation of whatever content it describes. Like any
representation, it selects and summarizes, reducing the content
of the media itself to a more compact and easily manipulable
form. This manipulability is the raison d’etre for the metadata in
the first place.
One important property of representations is that they are
artifacts and especially purposive artifacts. This means that
every representation has a set of associated purposes and the
representation’s criteria of selection and summarization reflect
those purposes. For instance, the metadata associated with
photographs in a news organization might be very different from
the metadata associated with photographs by a fashion company
or by an advertising agency.
The multiplicity of purposes leads directly to a need for multiple
descriptions or, technically, multiple schematizations or

Figure 1: The Metadata Twist
205