R. Vuorikari, N. Manouselis, E. Duval, “Using Metadata for Storing, Sharing, and Reusing Evaluations in Social
Recommendation: the Case of Learning Resources”, accepted for publication in Go D.H. & Foo S. (Eds.) "Social Information
Retrieval Systems: Emerging Technologies and Applications for Searching the Web Effectively", Hershey, PA: Idea Group
Publishing.

Using Metadata for Storing, Sharing and Reusing Evaluations for
Social Recommendations: the Case of Learning Resources

Riina Vuorikari
1
, Nikos Manouselis
2
, Erik Duval
1

1
Department of Computer Science,
Katholieke Universiteit Leuven (K.U.Leuven)
B-3001 Leuven, Belgium
{Riina.Vuorikari,Erik.Duval}@cs.kuleuven.be
2
Informatics Laboratory,
Div. of Informatics, Mathematics & Statistics,
Dept. of Science, Agricultural University of Athens
75 Iera Odos Str., 11855, Athens, Greece
nikosm@aua.gr


Abstract

Social information retrieval systems, such as recommender systems, would benefit from sharable and
reusable evaluations of online resources. For example, in distributed repositories with rich collections
of learning resources, users may benefit from evaluations (ratings, reviews, annotations, etc.) that
previous users have provided. Moreover, sharing evaluations could help attain the critical mass of data
required for social information retrieval systems to be effective and efficient. This kind of
interoperability requires a common data model that could be used to describe in a reusable manner
which evaluation approach has been applied, as well as the results of that evaluation. In this chapter,
we discuss this need, focusing on the rationale for a data model that can be used to facilitate the
representation, management and reuse of evaluation results of learning resources in an interoperable
manner. For this purpose, we review a variety of evaluation approaches for learning resources, and
study ways in which evaluation results may be characterised, so as to draw requirements for sharable
and reusable evaluative metadata. Usage scenarios are discussed that illustrate how evaluative metadata
could support recommender systems.

Keywords: Evaluation, metadata, learning objects, interoperability.


1. Introduction

Internet users are often times overwhelmed by the flow of online information, hence the need
for adequate systems that will help them manage such situations (Hanani et al., 2001).
Recommender systems attempt to guide the user in a personalised way to interesting and
useful items in a large space of possible options by producing individualised
recommendations as output (Burke, 2002). They are usually classified into two basic types
according to how recommendations are produced (Adomavicius & Tuzhilin, 2005): content-
based recommendation, where a user is recommended items similar to the ones that she
preferred in the past; and collaborative recommendation (or collaborative filtering), where a
user is recommended items that people with similar tastes and preferences liked in the past. A
recommender system requires a description of either the characteristics of the resources (for
content-based recommendation), or of user preferences for resources in the form of
evaluations or ratings (for collaborative recommendation).

There is an abundance of real-life applications of recommender systems on the Web that
provide users with personalised recommendations regarding online content and services
(Miller et al., 2004). In some application domains, the information used as input for
recommendation (e.g. the description of resources or the evaluations provided by users) may
be reused between different user communities or different recommender systems. For
content-based systems, this can be achieved when standardised descriptions of the resources
are used as input. For instance, e-commerce recommender systems could potentially be built
upon standardised frameworks for the description of the recommended items, such as the
UN/CEFACT UNSPSC catalogue of product and services classification
1

R. Vuorikari, N. Manouselis, E. Duval, “Using Metadata for Storing, Sharing, and Reusing Evaluations in Social
Recommendation: the Case of Learning Resources”, accepted for publication in Go D.H. & Foo S. (Eds.) "Social Information
Retrieval Systems: Emerging Technologies and Applications for Searching the Web Effectively", Hershey, PA: Idea Group
Publishing.

(http://www.unspsc.org/). In e-learning recommender systems, interoperability of content-
based recommender systems can be facilitated by existing technologies as well. For example,
characteristics of digital learning resources may be described by using metadata standards
such as the IEEE Learning Object Metadata standard (IEEE LOM, 2002).

However, reuse and shareability of user feedback (such as user opinions, ratings and reviews)
has not been on the focus of discussion for recommender systems. More specifically, in the
case of collaborative recommendation, there are currently no proposals of frameworks or
schemas for storing, sharing and reusing evaluations of resources in a common data format.
Such a framework could work to facilitate the reuse and interoperability in several domains,
as the learning technologies' one. This chapter focuses on the case of evaluation approaches
for digital learning resources and aims to point out that there is an opportunity to reuse
evaluative metadata. This chapter attempts to describe the issues and work towards possible
leads to solve the problematic in the future, rather than trying to give one oversimplified
answer to it.

The structure of this chapter is as following: first we provide the background to previous work
that introduces the use of metadata for digital learning resources and for storing information
about the evaluation/quality of digital learning resources. Then, a review of a sample of
current approaches used for evaluation of learning resources is carried out. In addition, a
tentative classification of evaluation approaches is performed, and their produced evaluation
results are studied. Furthermore, the chapter proposes a rationale and need for defining a
reusable and interoperable model to store approaches and their results in evaluative metadata,
and discusses the benefits of reusing evaluative results in the context of recommender
systems. Characteristic scenarios of such potential application to support interoperable social
recommendation are given. Finally, the conclusions of this study and the directions of future
work are provided.


2. Related Work

2.1 Metadata

Metadata is defined as structured information that describes, explains, locates, or otherwise
helps retrieving, using, or managing a resource. It is often called ‘data about data’ or
‘information about information’ (NISO, 2004; Steinacker et al., 2001; Duval et al., 2002).
Metadata contains data items that can be added to or attached to a resource. Much of the more
“traditional” work focuses on direct human interaction with metadata, listing the creator,
subject, title, and other data needed to find and manage the resource. However, we believe
that the focus should be more on less direct approaches, that “hide everything but the
benefits” (Duval et al. 2004).

Shreeves et al. (2006) discuss the qualities of sharable metadata in the context of digital
libraries. They argue that metadata should not only be useful in its local context, but also
usable to services outside of the local context in order to support search interoperability, i.e.
being sharable. It would be reasonable to apply this not only to the metadata that describe the
object (descriptive metadata), but to the metadata that is generated by users in the format of
annotations, reviews and ratings. Some early standardisation work has taken place in this area
already in 1997 when the World Wide Web Consortium PICS-specification enabled first- and
third-party rating of content to give users maximum control over the content they receive
without requiring new restrictions on content providers. This application of content rating
further motivated the development of RDF and served as a start for the work on the
“Semantic Web” and its many tools (Oram, 2001).

2.2 Quality and Evaluative Metadata
2