Usability Evaluation of Learning Object Indexation: The
ARIADNE Experience
Jehad Najjar, Joris Klerkx, Stefaan Ternier, Katrien Verbert, Michael Meire and
Erik Duval
Computer Science Dept., Katholieke Universiteit Leuven, Belgium
Jehad.Najjar@cs.kuleuven.ac.be
Joris.Klerkx@cs.kuleuven.ac.be
Stefaan.Ternier@cs.kuleuven.ac.be
Katrien.Verbert@cs.kuleuven.ac.be
Michael.Meire@cs.kuleuven.ac.be
Erik.Duval@cs.kuleuven.ac.be
Abstract: This paper investigates usability problems of indexation tools for Learning Objects. Such tools
are used to describe and deploy learning objects with their metadata in Learning Object Repositories
(LOR’s). Metadata is provided to facilitate the search and management of learning objects. The complexity
of manually indexing learning objects results in a bottleneck for the introduction of such objects (Duval and
Hodgins, 2003), (Kabel, et al., 2003). In this paper, we present the results of a usability evaluation aimed at
determining some usability issues in two ARIADNE indexation tools. Results show that metadata
semantics is significantly influenced by usability perspectives (such as the user interface, functionalities
provided, domain knowledge, etc). The findings and recommendations of this study are generalised for
other indexation tools.
Keywords: Metadata, User Studies, Learning Object Indexation, Learning Objects
1. Introduction
In recent years, a lot of attention has been given to the development of tools that facilitate
indexation and retrieval of learning materials (learning objects) in Learning Object Repositories
(LORs) (Duval and Hodgins, 2003), such as (Ariadne, 2004), (Merlot, 2004) and (SMETE,
2004). However, little attention has been given to the usability evaluation of these developed
tools (Duval and Hodgins, 2003). This evaluation will help researchers to determine to what
extent indexation tools enable users to reach their goals effectively and efficiently (Jones, et al.,
1998), (Marchionini, et al., 2000). In addition to this, it will help us to determine users’
satisfaction on the overall use of the evaluated tools. Furthermore, usability evaluation for
indexation tools allows us to understand how real users interact with such tools. For example,
we can obtain the average time needed to perform a particular task, the level of satisfaction,
and the ease of use of the indexation tool. Hereby, findings and recommendations obtained
from such usability evaluations help to improve the indexation of learning objects.
Several usability studies have been made in the context of Internet browsing (Cockburn and
McKenzie, 2000) (Nielsen, 2004), digital libraries (Jones, et al., 1998) and other computer
applications. On the other hand, this issue has not been studied intensively yet in the context of
LORs (Duval and Hodgins, 2003). In a previous work (Najjar, et al., 2003) we investigated the
behavior of users who index relevant learning objects in the KPS, by analysing the usage logs.
In this paper, we show results of the usability evaluation applied to two Ariadne indexation tools
(SILO and Toledo, see section 2). Both tools are deployed to the Ariadne Knowledge Pool
System (KPS).
Results of this study show that indexation of learning objects may be influenced by different
usability perspectives. Firstly, interface of indexation tools. Secondly, functionalities provided to
facilitate the indexation process, such as automatic indexation. Thirdly, indexer domain
knowledge about the introduced learning objects. In order to collect high quality metadata for
learning objects, we provide the following recommendations. First, the user interface of
indexation tools should be simple and adapted to the user, not to the metadata standard used
by the LOR. Second, functionalities such as automatic indexation are helpful and it should use
intelligent techniques to extract metadata of high quality.
281

3rd European Conference on e-Learning
The paper is structured as follows: in section 2, the two evaluated Ariadne clients are
introduced. Methods and procedures for the usability evaluation are introduced in section 3. In
section 4, the results obtained from the study are presented. Discussion on the results is given
in section 5. Finally, conclusions and future work are given in section 6.
2. Ariadne clients
Ariadne (ARIADNE, 2004) is a European project started in the year 1996. It is an integrated E-
Learning environment, incorporating a Knowledge Pool System (KPS), a web-based course
environment and a number of content authoring tools. The KPS is a Learning Object Repository
(LOR), similar to (Merlot, 2004), (SMETE, 2004), etc. The main goal of Ariadne is to foster the
share and reuse of learning objects. In order to achieve that goal an Indexation- and Query tool
has been developed to enable users’ index and search relevant learning objects in the KPS.
The indexation- and query tool uses 47 metadata elements to describe learning objects, from
which 18 are mandatory (Neven, et al., 2003). The metadata elements form what is called a
metadata Application Profile of the IEEE Learning Object Metadata (LOM) standard (IEEE,
2002). An early version of Ariadne metadata elements in conjunction with the IMS (IMS, 2004)
elements set formed the backbone of the IEEE LOM.
2.1 SILO client
Figure 1 shows the current version of the Indexation- and Query tool which is called SILO
(Search and Index Learning Objects). SILO is proposed to enhance the indexation and search
procedures in the Ariadne KPS by providing a number of functionalities (search functionalities
are not relevant for this paper):
ƒA profile (template) of metadata elements: elements provided for this profile are
automatically presented to the user when inserting new learning objects.
ƒAutomatic Indexation: a number of metadata elements are automatically extracted from the
learning object itself. For example, the document title, file name, required disk space, MIME
type, etc. are automatically presented to indexers.
Figure 1: Screenshot of SILO indexation client
2.2 Toledo client
Figure 2 shows the evaluated version of Toledo client. Toledo is an Ariadne building-bock for
Blackboard e-Learning management system. Toledo is used in K.U.Leuven University to
introduce learning objects to the Ariadne KPS. Toledo is different from SILO in the following
issues:
ƒToledo has a different look-and-feel than SILO.
ƒMore elements are automatically indexed in Toledo than SILO does (based on the narrow
context, see next point).
ƒToledo is used only in the context of K.U.Leuven. Only K.U.Leuven instructors are able to
introduce or search learning objects using it. SILO is of more general context. Users from
different institutions and countries, also, of different education levels can use it.
282