1RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu
Abstract
The aim of the open educational resource (oER) development movement is to provide free access
to high-quality educational materials in repositories. However, having access to a large amount of
educational materials does not provide any assurance of their quality, and the mechanisms so far
used to recommend educational resources have shown themselves to be lacking for a variety of
reasons. Most evaluation systems are based on a costly manual inspection, which does not allow
all materials to be evaluated. Moreover, it is often the case that other useful pieces of information
are ignored, such as the use that users make of the materials, the evaluations that users perform on
Ascertaining the Relevance
of Open Educational Resources
by Integrating Various Quality
Indicators
ARTICLE
Javier Sanz Rodríguez
javier.sanz.rodriguez@uc3m.es
Temporary Part-time lecturer, Department of Informatics, Carlos III University Madrid
Juan Manuel Dodero Beardo
juanma.dodero@uca.es
Tenured University lecturer, Department of Computer languages and Systems, University of Cadiz
Salvador Sánchez Alonso
salvador.sanchez@uah.es
Tenured University lecturer, Department of Computer Science, University of Alcalá de Henares
Submitted in: october 2010
Accepted in: May 2011
Published in: July 2011
Recommended citation
SANZ, Javier; DoDERo, Juan Manuel; SÁNCHEZ, Salvador (). “Ascertaining the Relevance of open Edu-
cational Resources by Integrating various Quality Indicators” [online article]. Revista de Universidad y So-
ciedad del Conocimiento (RUSC). vol. , No . UoC. [Accessed: dd/mm/yy].
<http://rusc.uoc.edu/ojs/index.php/rusc/article/view/vn-sanz-dodero-sanchez/vn-sanz-dodero-
sanchez-eng>
ISSN -X

2RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
them and the metadata used to describe them. To try and improve this situation, this article presents
the shortcomings of existing proposals and identifies every possible quality indicator that is able to
provide the necessary information to enable materials to be recommended to users. By studying
a significant set of materials contained in the MERloT repository, the relationships among various,
currently available quality indicators were analysed and numerous correlations among them were
established. on the basis of that analysis, a measure of relevance is proposed, which integrates all
existing quality indicators. Thus, the explicit evaluations made by users or experts, the descriptive
information obtained from metadata and the data obtained from the use of the latter are employed
in order to increase the reliability of recommendations by integrating various quality aspects. In
addition, this measure is sustainable because it can be calculated automatically and does not require
human intervention; this will allow all educational materials located in repositories to be rated.
Keywords
relevance, open educational resources, MERloT, e-learning
Determinando la relevancia de los recursos educativos abiertos
a través de la integración de diferentes indicadores de calidad
Resumen
El propósito del movimiento de desarrollo de recursos educativos abiertos es proporcionar libre acceso a
materiales educativos de alta calidad disponibles en repositorios. Sin embargo, tener acceso a una gran
cantidad de materiales educativos no garantiza que estos sean de calidad, y los mecanismos empleados
para recomendar los recursos educativos utilizados hasta la fecha se han mostrado insuficientes por dife-
rentes motivos. La mayoría de los sistemas de evaluación están basados en una costosa inspección manual
que no permite tener evaluados todos los materiales; además, muchas veces no se tienen en cuenta otras
informaciones útiles como la utilización que hacen los usuarios de los materiales, las evaluaciones hechas
por los usuarios y los metadatos que describen el material educativo. Para intentar mejorar esta situación,
en este documento se exponen las carencias de las propuestas existentes y se identifican todos los posibles
indicadores de calidad que pueden aportar información sobre qué materiales recomendar a los usuarios.
A través del estudio de un conjunto significativo de materiales del repositorio Merlot se analizan las relacio-
nes existentes entre los distintos indicadores de calidad disponibles, para constatar que existen numerosas
correlaciones entre ellos. Posteriormente y a partir de este análisis, se propone una medida de relevancia
que integre todos los indicadores de calidad existentes. De esta manera se utilizarán las evaluaciones ex-
plícitas realizadas por usuarios o expertos, la información descriptiva proveniente de los metadatos y los
datos que proceden del uso de estos, para lograr aumentar la fiabilidad de las recomendaciones al integrar
diferentes perspectivas de la calidad. Además, como esta medida se puede calcular de forma automática
se garantizará su sostenibilidad, ya que no necesitará de la intervención humana para su cálculo, lo que
permitirá que todos los materiales educativos ubicados en repositorios estén valorados.
Palabras clave
relevancia, recursos educativos abiertos, Merlot, e-learning

3RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
1. Introduction
our knowledge society demands competencies and skills that require the use of new educational
practices, such as the use of open educational resources (oERs) available on the Internet (Schaffert
& Geser). In a similar way to open software development, with projects such as linux or Apache,
the world of education is trying to develop high-quality oERs with rights that allow users to reuse
them and adapt them to suit their respective contexts (Kelty et al.). However, as is frequently the
case for any resource searching task, most searches in repositories return a vast number of materials,
thus making it difficult for users to decide which of them are best suited to their needs. Without a
formalised process that allows an algorithm to calculate the relative importance of the resources,
most materials searches will be lacking and their usefulness limited (Brownfield & oliver). To try and
overcome this problem, most repositories have used expert and user evaluations of educational
materials. Specifically, Tzikopoulos et al. identified that, of the 59 repositories contemplated in their
study, 23 offered various mechanisms for evaluating educational materials. However, the evaluation
system used so far is lacking (Kelty et al.) for a variety of reasons.
The task of manually reviewing materials is costly, and the amount of educational materials
is enormous and growing by the day. For example, at the time of the study (october 2009),
there were 21,399 materials in the MERloT repository, of which just 2,867 (13%) had been peer
reviewed. Consequently, unevaluated materials appear at the end of search results, as if they were
poor-quality resources. This situation has arisen because existing evaluation initiatives use a costly
inspection of the materials as the main source of information. But, as ochoa and Duval point out, for
a measure of oER quality to be useful, it needs to be calculated automatically. Furthermore, when
analysing the reliability of these explicit evaluations, we find that there are a number of problems.
Most expert evaluations are performed individually, which represents a limitation on their validity.
To overcome this limitation in part, it would be necessary to develop collaborative evaluation
processes in the repositories, and this would increase the cost of evaluating resources even more
(Boskic). Regarding user reviews, we also find that there are severe limitations on them for a variety
of reasons, such as the lack of user training, the potential subjectivity of tastes, etc. (Han). Moreover,
only a small number of users provide these evaluations and, as a result, their evaluations may not
be representative of the opinions of all users as a whole (Kay & Knaack). Along similar lines, Akpinar
performed a validation study on certain evaluation areas of the learning object Review Instrument
(loRI). The study compared evaluations with student and lecturer surveys and concluded that loRI
evaluations were not sufficient to predict the educational benefits that might be obtained from
oERs.
In addition, while there are various initiatives that allow a search to be performed across several
repositories, such as the EduSource project (McGreal et al.), we find that repositories have different
evaluation systems, thus making it difficult to sort the results returned for several repositories. In a
similar way to the various metadata application profiles, it is crucial to develop strategies that allow
different repository evaluation systems to be integrated (li et al.).
Moreover, Kelty et al. assert that educational resources are being evaluated statically, just like

4RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
traditional educational materials used to be. To overcome this deficiency, they propose that
evaluations should not only focus on content, but also contemplate potential contexts of use.
In any event, the availability of large databases of data with evaluations has opened up new
opportunities for developing indicators that could complement existing evaluation techniques
based on an enormous effort of manually inspecting materials. Indeed, such evaluation techniques
could be replaced by other measures that are automatically calculated, thus facilitating an indicator
of the quality of educational materials in a less costly way (García-Barriocanal & Sicilia).
A potential improvement that Kelty et al. propose is to use systems similar to the lens mechanism
that the Connexions repository uses, in which each lens is created by applying an evaluation
criterion to materials, including peer reviews, popularity, number of re-uses, number of times they
are linked, etc.; the application of one lens or a combination of lenses allows educational materials
to be filtered.
In a similar way, Han indicates that the current systems for recommending educational materials
lack a weighting mechanism that would otherwise allow the evaluative data from various sources to
be taken into account, since each one provides information differently. Consequently, he proposes
an integrated quality indicator that combines explicit expert and user evaluations, anonymous
evaluations and implicit indicators (favourites and retrievals).
Drawing our inspiration from the last two proposals, the aim of this study is to formulate a
relevance indicator that: can be calculated automatically; ensures that all resources are rated; and
encompasses available quality indicators, which can be classified into three categories:
• Evaluative. This encompasses all explicit expert and user evaluations.
• Empirical. This refers to information on materials usage, as obtained from their implicit data,
such as retrievals, the number of users who bookmark them in their favourite materials lists,
etc.
• Characteristic. This refers to descriptive information on the characteristics of the materials, as
obtained from their metadata.
The rest of this article is structured as follows: in sections 2, 3 and 4, the quality indicators are
identified and grouped under the categories referred to earlier; in section 5, an analysis of the
relationships among the quality indicators is performed by studying a significant set of materials in
the MERloT repository; in chapter 6, a measure of relevance is proposed and applied to the set of
materials under investigation; and finally, in chapter 7, the conclusions are drawn.
2. Evaluative Quality Indicators
There are many studies on how to evaluate oERs, such as those proposed by Kay and Knaack and by
Kurilovas and Dagiene; the evaluations that have been put into practice are those implemented in
the various repositories.

5RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
In the MERloT repository, materials are evaluated by a peer-review process that focuses on three
aspects: content quality, ease of use and potential effectiveness as a teaching-learning tool; each
aspect is rated from 1 to 5 (poor to excellent). The weighted mean of the three aspects becomes the
educational resource’s final evaluation value. Registered users can also evaluate and comment on
resources.
The elera repository allows users to evaluate materials by using loRI, which focuses on nine
aspects: content quality, learning goal alignment, feedback and adaptation, motivation, presentation
design, interaction usability, accessibility, reusability and standards compliance. In a similar way to
MERloT, each aspect is rated on a scale from 1 to 5. Worthy of note is that collaborative evaluation
initiatives have been developed through elera, in which groups of experts participate. When this
approach is taken, materials are first evaluated individually and asynchronously, and then the
evaluations are discussed prior to agreeing on a final rating.
Finally, the Connexions repository proposes a quality evaluation by using a lens mechanism; the
application of one lens or a combination of lenses allows users to filter materials to obtain the most
suitable ones. Among potential lens types are those based on peer reviews and those elaborated by
users (Baraniuk).
3. Empirical Quality Indicators
When it comes to recommending resources, the use of implicit data resulting from usage is an idea
that has already been applied to Web page selection. Along these lines, Claypool et al. show that
it is worthwhile using implicit data obtained from user behaviour for sorting search results. These
measures have been used to improve searches on the Web, since they reflect the users’ interests and
degrees of satisfaction, and are less costly than explicit evaluations (Fox et al.).
In the particular case of oERs, implicit information about resource retrieval or bookmarking in
favourites is available in the MERloT repository. In Connexions, lenses for recommending materials
can be created automatically on the basis of data such as popularity, number of re-uses, number
of times they are linked, etc. (Baraniuk). Building on this idea, Kumar et al. propose that, besides
the evaluations available in the repositories, data on materials usage could be used to supplement
information on the quality of educational materials. Similarly, Yen et al. propose using information on
references to educational materials so as to sort them using the Page Rank algorithm that Google
uses to return search results.
likewise, in this section we could include social tagging systems, which are a basic way of adding
descriptive metadata to educational content. While social tagging tools have received a great deal of
criticism due to their terminological imprecision (Cueva & Rodríguez), there are some proposals that
suggest using this information to build a recommendation metric like, for example, counting each
tag as a vote for the educational resource (Yen et al.).

6RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
4. Characteristic Quality Indicators
The characteristic category encompasses indicators based on metadata, which can take advantage
of the potential of information describing an educational resource. Along these lines, various
authors have proposed their own indicators: ochoa and Duval propose using metadata to sort
the search results for educational materials and to be able to recommend the most suitable ones.
Specifically, they propose a set of relevance metrics for educational materials, applying the same
ideas used to classify Web pages, scientific articles, etc. Knowing which materials are the most
relevant from different viewpoints would make it easier to choose an educational resource for
re-use. The information for estimating these relevance metrics is obtained from data on users’
retrieval of educational materials, the metadata on the materials, registers of materials usage and
information on the context. Zimmermann et al. remind us that, in order to reuse an educational
resource designed for a specific context, it is often necessary to adapt it to the new context in
which it will be used. Consequently, they propose evaluating the adaptation effort required in order
to reuse it. Adaptation to a new learning context may involve such tasks as: adapting materials to
a new learning objective or a new group of students (different from the target group for which
they were originally designed); extracting part of the content from the resource; and combining
the resource with other educational materials. When faced with the question about how to find
learning materials that can be adapted to a new context in the least costly way, Zimmermann et al.
propose measuring metadata similarities to ascertain adaptation needs. Finally, Sanz et al. propose
metadata-based reusability metrics – the calculation of which can be automated – that measure
aspects such as consistency and educational and technological reusability, thus allowing materials
with greater potential for re-use to be chosen.
5. Analysis of the Correlations
among the Various Quality Indicators
once the various quality indicators have been identified and grouped under the categories referred
to earlier, the relationships among them can be analysed. It should be stipulated that the study was
conducted on a set of 141 materials selected from MERloT, the repository from which we were able
to obtain indicators for all the categories. This set of materials was retrieved on 1 october 2009. It
included all materials added to the repository between 2005 and 2008 that had been evaluated by
experts and had received comments from users. Table 1 shows the indicators chosen for the study:
Personal Collections indicates the number of times a resource has been bookmarked in favourites;
Exercises are teaching proposals that link to one or several materials; and Used in Classroom indicates
whether a resource has been used in the classroom by the user evaluating it. Regarding the indicator
based on metadata, the Reusability indicator proposed by Sanz et al. was used.

7RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
Table 1. Quality indicators studied
Evaluative Empirical Characteristic
Overall Rating
Content Quality
Effectiveness
Ease of Use
Comments
Personal Collections
Exercises
Used in Classroom
Reusability
Then the correlations among the indicators of the various categories were studied. Table 2 shows
that there is a strong correlation among the explicit ratings given by experts. However, there is hardly
any correlation among the ratings given by users. only ease of use is correlated with the ratings
given by experts. This may be due to the fact that users do not have the necessary knowledge to
evaluate the resource they are analysing, perhaps because it falls within an area or level beyond
their scope. It may also be due to the fact that users place greater importance on ease of use in their
overall evaluation of educational materials. In this respect, Han points out that it is difficult to place
a numeric value on users’ tastes in a quality evaluation. For example, if users prefer certain types of
literature, they are more likely to rate educational materials dealing with them more highly.
Table 2. Kendall’s Tau correlation among explicit ratings
Overall Rating Content Quality Effectiveness Ease of Use Comments
Overall Rating 1 0.776** 0.718** 0.663** 0.096
Content Quality 0.776** 1 0.724** 0.615** 0.107
Effectiveness 0.718** 0.724** 1 0.507** 0.126
Ease of Use 0.663** 0.615** 0.507** 1 0.172*
Comments 0.096 0.107 0.126 0.172* 1
** Correlation is significant at 0.01 * Correlation is significant at 0.05
Table 3 illustrates the correlations among indicators in the Evaluative and Empirical categories,
and shows a correlation between the materials in Personal Collections and expert evaluations.
Table 3. Kendall’s Tau correlation between explicit and empirical ratings
Personal Collections Exercises Used in Classroom
Overall Rating 0.171** 0.033 0.045
Content Quality 0.145* -0.014 0.034
Effectiveness 0.224** 0.047 0.123
Ease of Use 0.146* 0.036 0.071
Comments 0.046 -0.007 0.049
** Correlation is significant at 0.01 * Correlation is significant at 0.05

8RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
Table 4 shows the correlations among the indicators in the Empirical category.
Table 4. Kendall’s Tau correlation among empirical ratings
Personal Collections Exercises Used in Classroom
Personal Collections 1 0.227** 0.105
Exercises 0.227** 1 0.298**
Used in Classroom 0.105 0.298** 1
** Correlation is significant at 0.01 * Correlation is significant at 0.05
Finally, Table 5 shows the correlations with the metadata-based Reusability indicator.
Table 5. Kendall’s Tau correlations with the metadata-based Reusability indicator
Reusability
Personal Collections 0.240**
Exercises 0.062
Used in Classroom 0.092
Overall Rating 0.287**
Content Quality 0.301**
Effectiveness 0.300**
Ease of Use 0.279**
Comments 0.031
** Correlation is significant at 0.01
The correlations found among the indicators of the various categories support the idea that they
are all measures of quality obtained from different viewpoints, and that they can be complemented
to obtain an indicator that rates the relevance of an oER.
6. Integrating Quality Indicators
into a Measure of Relevance
The measure of relevance combines all information on the quality of a resource. Consequently, if
a quality indicator is missing, a measure of relevance can be obtained from existing indicators and
calculated automatically. This will solve the current problem whereby materials without expert
evaluations appear at the end of any search, automatically ruling them out. It will also increase
the reliability of recommendations. The relevance of a learning resource called o is described in
(1).

9RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
1 1 1
( ) ( ) ( ) ( )
n m l
i j ki j k
i j k
Relevancia o Valorativa o Característica o E mpírica oa b g
= = =
= + +∑ ∑ ∑
Here, , ,i j ka b g represent the weights of the various Evaluative, Characteristic and Empirical
relevances, and n, m and l indicate the number of indicators in each quality category. In addition, all
the relevances are normalised in a range of values from 0 to 5, which is the scale used for MERloT’s
evaluative indicators, and their mean values are obtained when several data are available. If one of
the data is missing, the weights are adjusted so as not to penalise its absence from the calculation of
relevance, and the equation described in (2) will always be fulfilled:
Adapting the generic formula to the specific case of the MERloT repository study, it is possible to
explain how 1 2,a a are the weights of the two Evaluative indicators (overall rating and comments)
and 1 2( ) , ( )Valorativa o Valorativa o are the mean values of the two Evaluative indicators of learning
resource o .
To ascertain the weights, two sources of information were used. First, the weights proposed by
Han for integrating the various measures of quality into the rating of educational materials, and
second, information obtained in the previous section on the studies of correlations among quality
indicators. By combining both sources of information, the resulting final model is expressed in Table
6.
Table 6. Weightings of the quality indicators studied
Category Indicator Weight
Evaluative Overall Rating
Comments
0.3
0.1
Empirical Personal Collections
Exercises
Used in Classroom
0.15
0.1
0.05
Characteristic Reusability 0.3
To explain the use of the relevance indicator, we studied the Graph Theory lessons educational
resource available in the MERloT repository. Table 7 shows the values of the quality indicators
obtained at the time the study was conducted.
To integrate all of these values into the final formula, we had to perform a transformation of the
usage indicators (Personal Collections, Exercises and Used in Classroom).
First, we had to normalise them, taking account of the amount of time the resource had been
available in the repository. In this instance, the resource had been available since 24 September
2005. obviously, a resource that has been available for a longer period of time may have been used
(1)
1 1 1
1
n m l
i j k
i j k
a b g
= = =
+ + =∑ ∑ ∑ . (2)
Evalu ti e haracteristic i l]Relevance
Evalu e Evaluative

10
RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
more often, hence the need to normalise this value. In addition, the Empirical indicators had to be
normalised on the reference scale used (0 to 5). This indicated that materials with a relevance value
closer to 5 are more relevant.
Table 7. Quality indicator values for the Graph Theory Lessons educational resource
Graph Theory Lessons
Personal Collections 9
Exercises 0
Used in Classroom 3
Overall Rating 5
Comments 4
Reusability 4.49
Second, the relevance indicator was applied to the set of materials obtained from the Merlot
repository and studied. Figure 1 shows the statistical distribution of the measure of relevance
compared to a normal distribution. This graph allows us to illustrate that the measure of relevance
has a distribution in which a minority of materials had low or very low ratings, and the majority
had intermediate values. This behaviour may correspond to the one that is expected in a process of
educational materials evaluation.
Figure 1. Relevance indicator histogram
Fr
eq
ue
nc
y
Normal
Distribution
Relevance
7. Conclusions
The correlations found among the indicators of the various categories support the idea that they
are all measures of quality obtained from different viewpoints and that they can be complemented
to obtain an indicator that rates the relevance of an educational resource. The use of this measure
of relevance may offer several advantages when it comes to selecting quality educational materials.
First, the main advantage is that will help the end-user select educational materials.

11
RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
Another advantage is that it will improve the reliability of evaluations, since it encompasses all
existing and relevant information: expert and user evaluations, usage data and information contained
in their metadata. Given the large number of educational materials available in repositories, being
able to provide a quality indicator that encompasses very diverse aspects – ratings by users with
different profiles, resource usage data and resource characteristics described in metadata – will help
to locate quality educational materials for re-use.
Finally, worthy of note is the advantage offered by the sustainability of the indicator’s calculation.
As the measure of relevance can be calculated automatically, it will allow all educational materials
available in repositories to have a rating, even when one of the quality indicators is missing. For
example, when a resource has been evaluated by users but not by experts, and data on their usage
and characteristics are available, the measure of relevance can be calculated automatically, thus
providing a recommendation to help users in the process of selecting materials.
References
AKPINAR, Y. (2008). “validation of a learning object Review Instrument: Relationship between Ratings
of learning objects and Actual learning outcomes”. Interdisciplinary Journal of Knowledge and
Learning Objects. vol. 4, pages 291-302.
BARANIUK, R. G. (2007). “Challenges and opportunities for the open Education Movement: A
Connexions Case Study”. In: Opening Up Education: The Collective Advancement of Education
through Open Technology, Open Content, and Open Knowledge. California: MIT Press. Pages 116-132.
BoSKIC, N. (2003). Faculty Assessment of the Quality and Reusability of Learning Objects. PhD thesis.
Alberta (Canada): Athabasca University.
BRoWNFIElD, G.; olIvER, R. (2003). “Factors influencing the discovery and reusability of digital
resources for teaching and learning”. In: G. Crisp, D. Thiele, I. Scholten et al. (eds.). Interact, Integrate,
Impact: Proceedings of the 20th Annual Conference of the Australasian Society for Computers in
Learning in Tertiary Education. Adelaide. pages 74-83.
ClAYPool, M.; lE, P.; WASED, M. et al. (2001). “Implicit interest indicators”. Proceedings of the
6th international conference on Intelligent user interfaces. New Mexico. Pages 33-40, doi:
10.1145/359784.359836.
CUEvA, S.; RoDRÍGUEZ, G. (2010). “oERs, Standards and Trends”. Revista de Universidad y Sociedad del
Conocimiento (RUSC). vol. 7, No 1. UoC.
FoX, S.; KARNAWATT, K.; MYDlAND, M. et al. (2005). “Evaluating Implicit Measures to Improve Web
Search”. ACM Transactions on Information Systems. vol. 23, No 2, pages 147-168.
GARCÍA-BARRIoCANAl, E.; SICIlIA, Miguel A. (2009). “Preliminary Explorations on the Statistical
Profiles of Highly-Rated learning objects”. Metadata and Semantic Research. Proceedings of Third
International Conference. Milan: MTSR 2009. Pages 108-117.
HAN, K (2004). Quality Rating of Learning Objects using Bayesian Belief Networks. PhD thesis. Canada:
Simon Fraser University.

12
RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
KAY, R.; KNAACK, l. (2009). “Assessing learning, quality and engagement in learning objects: the
learning object Evaluation Scale for Students (loES-S)”. Educational Technology Research and
Development. vol. 57, No 2, pages 147-168.
KAY, R.; KNAACK, l. (2007). “Evaluating the learning in learning objects”. Open Learning: The Journal of
Open and Distance Learning. vol. 22, No 1, pages. 5-28.
KElTY, C. M.; BURRUS, C. S.; BARANIUK, R. G. (2008). “Peer Review Anew: Three Principles and a Case
Study in Postpublication Quality Assurance”. Proceedings of the IEEE. vol. 96, No 6, pages 1000-1011.
KUMAR, v.; NESBIT, J.; HAN, K. (2005). “Rating learning object Quality with Distributed Bayesian
Belief Networks: the why and the how”. Fifth IEEE International Conference on Advanced Learning
Technologies. ICAlT 2005, pages 685-687.
KURIlovAS, E.; DAGIENE, v. (2009). “learning objects and virtual learning environments technical
evaluation criteria”. EJEL - Electronic Journal of e-Learning. vol. 7, No 2, pages 147-168.
lI, J. Z.; NESBIT, J. C.; RICHARDS, G. (2006). “Evaluating learning objects across boundaries: The semantics
of localization”. Journal of Distance Education Technologies. vol. 4, No 1, pages 17-30.
MCGREAl, R.; ANDERSoN, T.; BABIN, G. et al. (2004). “EduSource: Canada’s learning object Repository
Network”. International Journal of Instructional Technology and Distance Learning. vol. 1, No 3.
oCHoA, X.; DUvAl, E. (2006). “Quality Metrics for learning object Metadata”. Proceedings of World
Conference on Educational Multimedia, Hypermedia and Telecommunications. ED-Media 2006.
Pages 1004-1011.
oCHoA, X.; DUvAl, E. (2008). “Relevance Ranking Metrics for learning objects”. IEEE Transactions on
Learning Technologies. vol. 1, No 1, pages 34-48.
SANZ RoDRÍGUEZ J.; DoDERo, J. M.; SÁNCHEZ AloNSo, S. (2009). “A preliminary analysis of software
engineering metrics-based criteria for the evaluation of learning objects reusability”. International
Journal of Emerging Technologies in Learning. vol. 4, No 1, pages 30-34.
SCHAFFERT, S.; GESER, G. (2008). “open Educational Resources and Practices”. eLearning Papers. No 7.
Available at www.elearningpapers.eu. [Accessed: 26 october 2010].
TZIKoPoUloS, A.; MANoUSElIS, N.; vUoRIKARI, R. (2007). “An overview of learning object
Repositories”. In: Learning Objects for Instruction: Design and Evaluation. Idea Group Publishing.
Pages 44-64.
vUoRIKARI, R. Tags and self-organisation: a metadata ecology for learning resources in a multilingual
context. PhD thesis. open Universiteit Nederland.
YEN, N. Y.; HoU, F. F.; CHAo, l. R. et al. (2009). “Weighting and Ranking the E-learning Resources”.
Proceeding of the 9th IEEE International Conference on Advanced Learning Technologies. ICAlT 2009.
Pages 701-703.
ZIMMERMANN, B.; MEYER, M.; RENSING, C. et al. (2007). “Improving retrieval of re-usable learning
resources by estimating adaptation effort backhouse”. First International Workshop on Learning
Object Discovery and Exchange.

13
RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
About the Authors
Javier Sanz Rodríguez
javier.sanz.rodriguez@ucm.es
Temporary Part-time lecturer, Department of Informatics, Carlos III University Madrid
Javier Sanz Rodríguez holds a degree in Informatics awarded by the Technical University of Madrid
in . He also took a master’s degree in Science and Information Technology at Carlos III University
Madrid. In , he was awarded a doctorate in Informatics by the University of Alcalá de Henares. His
professional experience includes positions as a consultant for Everis and Deloitte & Touche, and as
an analyst for Telefónica Móviles. He is currently a temporary part-time lecturer at Carlos III University
Madrid, and a teacher of vocational Training in the Community of Madrid, where he participates in
projects connected with the use of ICTs in education.
Universidad Carlos III de Madrid
Av. Universidad, 
 leganés, Madrid
Spain
Juan Manuel Dodero Beardo
juanma.dodero@uca.es
Tenured University lecturer, Department of Computer languages and Systems, University of Cadiz
Juan Manuel Dodero holds a degree in Informatics awarded by the Technical University of Madrid in
 and a doctorate in Computer Engineering awarded by Carlos III University Madrid in . His
main fi elds of research are software and Web engineering, with a special focus on computer-assisted
learning applications. He has worked as an R&D engineer at Intelligent Software Components S. A.
and as a lecturer at Carlos III University Madrid. Since , he has been a tenured university lec-
turer at the University of Cadiz, Spain. He is the co-author of  publications in indexed international
journals, fi ve book chapters and more than  papers for international conferences on informatics
research. He is a founding member of, and sits on, the Management Committee of the Spanish Sec-
tion of the ACM Special Interest Group on Computer Science Education (SIGCSE). In , he was
awarded the young researcher distinction by the IEEE Technical Committee on learning Technology
for his contributions to research in this fi eld in the initial post-doctoral phase.
Universidad de Cádiz
C/ Chile, s/n
 Cadiz
Spain

The texts published in this journal are – unless indicated otherwise – covered by the Creative Commons
Spain Attribution . licence. You may copy, distribute, transmit and adapt the work, provided you attribute it
(authorship, journal name, publisher) in the manner specifi ed by the author(s) or licensor(s). The full text of the
licence can be consulted here: http://creativecommons.org/licenses/by/./es/deed.en.
14
RUSC vol. 8 No 2 | Universitat oberta de Catalunya | Barcelona, July 2011 | ISSN 1698-580X
CC Javier Sanz, Juan Manuel Dodero and Salvador Sánchez
http://rusc.uoc.edu Ascertaining the Relevance of open Educational Resources…
Salvador Sánchez Alonso
salvador.sanchez@uah.es
Tenured University lecturer, Department of Computer Science, University of Alcalá de Henares
Salvador Sánchez Alonso holds a qualifi cation as an Informatics engineer awarded by the Pontifi cal
University of Salamanca in  and a doctorate in Informatics awarded by the Technical University of
Madrid in . He worked as an adjunct lecturer at the Pontifi cal University of Salamanca from 
to  and from  to . From  to , he worked as a software engineer for Misys Inter-
national Banking Systems, a development company located in london. Since , he has been a
lecturer in the Department of Computer Science at the University of Alcalá de Henares He is currently
working in the fi eld of research, with a special focus on reusability and learning object metadata, the
semantic Web and software engineering.
Universidad de Alcalá de Henares
Ctra. Barcelona, km 
 Alcalá de Henares, Madrid
Spain