Stepping out of the box. Towards analytics
outside the Learning Management System
Abelardo Pardo and Carlos Delgado Kloos
Department of Telematic Engineering,
University Carlos III of Madrid, Spain
{abel,cdk}@it.uc3m.es
http://gradient.it.uc3m.es
Abstract. Most of the current learning analytic techniques have as
starting point the data recorded by Learning Management Systems (LMS)
about the interactions of the students with the platform and among
themselves. But there is a tendency on students to rely less on the func-
tionality oered by the LMS and use more applications that are freely
available on the net. This situation is magnied in studies in which stu-
dents need to interact with a set of tools that are easily installed on
their personal computers. This paper shows an approach using Virtual
Machines by which a set of events occurring outside of the LMS are
recorded and sent to a central server in a scalable and unobtrusive man-
ner.
Keywords: Learning analytics, Learning management system, virtual
machines
1 Introduction
The eld of Learning Analytics is emerging as a combination of business intel-
ligence, business logic, educational data mining and action analytics [8] where
data is collected, analyzed and interpreted to derive so called actuators to op-
timize a learning experience. Much the same way in which a regular web site
monitors the operations performed by the users to then infer patterns to suggest
or modify the experience of future users, in the context of learning, students
usually interact with a Learning Management System (LMS) that records all
the operations. This wealth of data can also be analyzed and transformed into
useful information to ultimately infer and apply changes to the current environ-
ment aiming at improving the process for the student, the instructor and/or the
institution.
There are numerous key developments that are behind the emergence of
this new eld. LMSs, both commercial and open source, include modules that
automatically register every event taking place in the platform. The higher the
percentage of course activity that takes place in the LMS, the more detailed
information is stored. For example, if a learning experience contains support for
on-line quizzes, formative assessment, a chat room and its own email service, the

2 A. Pardo, C. Delgado Kloos
platform may easily keep record of who did what, when and with whom. When
combined with the well established techniques in the area of web analytics,
having a detailed account of the interaction between students and instructors,
or among students becomes a reality.
Once the data is obtained from the tools used in a learning experience there
are multiple objectives that can be tackled. For example, The Signals project
a Purdue University [4] is an example of how learning analytics as described
in [10] are applied at an institutional level to create an early detection system for
student failure. The system uses the data already collected by the institutional
LMS to detect in real time the type of events that, based on previous information,
have a high probability of leading to student failure. Detecting these patterns
translates then into a set of measures that are taking to anticipate the problem
and thus reduce student failure rates.
Interaction has been shown to be an important factor in
uencing student
success [3]. The amount of interaction a student has with peers has a positive
correlation with the academic performance [12]. As a consequence, having a
detailed account of the interactions that occur in a learning experience will
likely oer a good predictor of academic performance, which itself is one of the
most important aspects of an educational institution. With these new tools, a
learning community can be \seen" in ways that were never considered before [7].
But in this scenario, there are several challenges that need to be overcome.
Campbell and Oblinger [5] characterized the process of learning analytics as
an engine with ve stages: capture, report, predict, act, and rene. The rst
step already faces the challenge of having an adequate observation capability.
A detailed account of any event that takes place in a learning scenario is the
rst requirement to have a solid foundation upon which to build the reporting,
predicting and acting mechanisms. After the data has been obtained, it should be
reported in meaningful forms to all stakeholders. Several visualization techniques
have been applied specically to data gathered in courses (see for example [14,
15]). The next challenge is to determine which factors are truly signicant to
achieve accurate predictions. In [13] a detailed analysis is performed considering
initially a set of 22 variables recorded by the LMS (BlackBoard Vistatm1. Out
of these 22 factors, only 13 were found to have a positive signicant correlation
with student nal grade. A nal multi-variable linear model is proposed with
only three of these factors accounting for 33% of the variability. These factors
are: total number of discussion messages posted, total number of mail messages
sent, and total number of assessments completed.
Once a model has been created, the following steps face the challenges of
inferring relevant interventions, and nally, to design a feedback process to rene
the overall mechanism. There are already several approaches that close this cycle.
In [11] a system is presented in which all the interactions of the students with
the course material and among each other are recorded and made available to
the instructors when modifying the course content. The system uses Semantic
1 www.blackboard.com

Stepping out of the box. Towards analytics outside the LMS 3
Web techniques to translate LMS logs into resource annotations that are then
inserted into the editing tool used by the instructors to create content.
1.1 The challenge of recording the interaction
But recording the interactions that take place in a learning environment is be-
coming more dicult. The initial model used during the early stages of LMS
deployment in educational institutions could be called \LMS-centric". There
were numerous analogies between LMSs and conventional knowledge manage-
ment tools. But with the advent of the Web 2.0, the \LMS-centric" model has
failed [6]. Although the latest LMSs oer an increasing set of features, students
are beginning to reach the educational institutions with solid experience on how
to interact with their peers in ways that are not covered by the LMS. The main
consequence is that a signicant part of the interaction during a learning expe-
rience is beginning to take part outside of the LMS in what it could be called a
\de-centralized approach".
Even the LMSs themselves have contributed to this de-centralization. For
example, most LMSs oer the option of receiving email notications when new
messages are posted in forums. The chances of students using an email client
outside the LMS are increasingly large. Email support is another example. LMSs
oer internal an email account to each user, but they are no competition in terms
of features other platforms available on the Net.
This tendency is more exacerbated when a learning experience contains a sig-
nicant amount of activities that cannot be embedded by any means in an LMS.
In experimental sciences, students typically require the use of special resources
for procedural activities. The extreme case of this tendency is in ICT education
where most of these special resources are applications that can be installed in
the student personal computer. Furthermore, some studies are beginning to con-
rm that students use conventional ICT tools to access an increasing number of
resources outside the institution LMS [21].
The main consequence of this tendency is that in order to maintain the
eectiveness of learning analytics, new techniques are required to extend beyond
the LMS-centric approach and adapt to the Web 2.0 style.
1.2 Observation to support assessment
Another factor that is changing the educational landscape is the transition from
a purely expository instructional method to a \learner-centered" approach [17]
where the tutor adopts a more supportive role, and the learner explores, partic-
ipates and is more active during the learning process. This change of philosophy
is having numerous ramications within the academic world. Entire degree pro-
grams are re-organized in order to accommodate the new role of the student.
Teaching sta needs to adapt their pedagogical techniques to a, sometimes, to-
tally new approach.
Together with these changes, numerous accreditation institutions have emer-
ged with the objective of assuring that educational institutions embrace quality

4 A. Pardo, C. Delgado Kloos
assurance and sustained innovation techniques. For example, ABET (Accredi-
tation Board for Engineering and Technology) is an institution that provides
accreditation for degrees in the area of applied science, computing and engi-
neering education. The focus of the accreditation process is on \what is learned
rather than what is taught"2.
The approach described in this document is being deployed in an engineering
degree that currently pursuing accreditation by ABET. The institution describes
what students are expected to know and be able to do by the time they graduate.
Again, in the specic case of engineering education, some of these outcomes have
a strong procedural nature. For example: \(k) an ability to use the techniques,
skills, and modern engineering tools necessary for engineering practice." [1].
In order to assure that on graduations students are capable of using modern
engineering tools, they need to practice with them through activities. This is
an example of the type of new outcomes that are being requested from applied
science degrees that are dicult to accommodate by conventional LMS. At most,
LMSs may cover this aspect of the learning process by supporting on-line quizzes,
but as an indirect measuring tool.
A second example of the limitations of LMSs is highlighted by another pro-
gram outcome: \(d) an ability to function on multi-disciplinary teams". Team-
work requires a high degree of student-student interaction. There are studies
that rely on interaction through forums hosted in an LMS to gain insight on the
level of collaboration within teams [2]. But if students are already used to com-
municate using a variety of Web 2.0 type of tools, it is highly unlikely that when
immersed in a collaborative setting, they would use an LMS for these tasks.
From the previous observations, there are several questions that lay ahead in
the area of learning analytics:
{ How much do they rely on interaction taking place in the LMS?
{ How can they cope with new forms of interaction?
{ How are they aected when analyzing interaction in collaborative environ-
ments?
This document describes the approach to obtain learning analytics in a con-
crete scenario of collaborative activities within a course of an engineering degree.
Although still in the preliminary stages, we believe there are several observations
that can help shed some light on the previous questions.
2 Approach
The approach was deployed in the face-to-face course \Systems Architecture",
which is part of the degree in Telecommunication Engineering 3 The total number
of students that initially signed for the course was 248 and were divided into ve
sections groups. The course contained the following learning outcomes:
2 www.abet.org
3 www.it.uc3m.es/labas/syllabus_en.html

Stepping out of the box. Towards analytics outside the LMS 5
1. Design and development of applications in the C Programming Language.
2. Use prociently the tools for application development.
3. Apply team working techniques to develop an application for a mobile device.
4. Use of self-learning techniques.
Outcomes 3 and 4 refer to generic methodological aspects. Team work was
used during the second half of the course (six weeks) in which groups of four
students were created by the instructors to work in a project. Several documents
about team dynamics were requested as readings and a class session was devoted
to discuss teamwork, agree on a team contract and discuss the dierent type
of con
icts that may arise. The measures to achieve outcome 4 were applied
throughout the entire course. Each session had two sets of activities, previous
and in-class. The set of previous activities required an objective that would be
reviewed in the following class. Students found this methodology signicantly
dierent to those used in other courses.
The course followed a continuous evaluation scheme. Five partial examina-
tions spread along the semester were combined with small exercise submissions.
The goal was to engage students to regularly work in the course. The nal course
grade was simply the sum of all these partial scores; no nal exam was given.
2.1 Providing a fully congured development environment
The main complication from the point of view of analyzing the interaction de-
rived from outcomes 2 and 3. In order assure that students use prociently the
tools for application development, they required a development environment
fully congured and, most importantly, with high availability (to promote o-
class work and do not overload computer rooms). This type of environment was
clearly beyond the reach of the institutional LMS, and therefore, the possibility
of observing the interaction with these tools was initially non-existent.
The adopted solution was based on the use of a virtual machine. Lately, vir-
tualization has been considered in education in order to easily facilitate students
fully congured machines that can execute with barely any conguration steps in
their personal computers [9]. The use of this this approach had several benets.
First, all students had initially the same exact set of tools properly congured
which greatly simplied the design of activities to use them. Second, the ma-
chine was congured so that students could access the les stored in their regular
personal computers. Third, the virtual machine (although including a fully con-
gured operating system) was portrayed to the students as the application to
use when working on the course material. And last, but most importantly, the
machine included a system to record the events occurring with respect to the
installed tools.
More precisely, the monitoring mechanism was capable of recording the fol-
lowing events:
{ Power-up and shutdown of the machine.
{ Invocation of a previously selected set of tools.

Stepping out of the box. Towards analytics outside the LMS 7
Fig. 1. Initial screen of the virtual machine
With the described conguration, students would see how the folder shown in
their desktop keeps changing its content. The type of resources that can be added
range from les (documents, audio, video) to URLs to access remote resources.
2.4 Encoding the events
The events recorded in the student machines oer a very detailed account of
the procedures followed as well as the tools that were invoked. The captured
information has been encoded using the CAM (Contextualized Attention Meta-
data) format [19, 22, 18]. CAM provides a data model for representing user activ-
ity together with contextual information. Educational application of the CAM
framework are discussed in [20], where the tool CAMera for monitoring and re-
porting on learning behavior is described. CAMera collects usage metadata from
diverse various applications, represent these metadata with CAM and reports
them to the learner. More complex applications in the scope of adaptation and
web-semantics have also been built based on this format [16, 16].
3 Initial Results
The initial objective of this approach is to explore ways to extend the data-
gathering phase of learning analytics beyond the LMS and into environments in

8 A. Pardo, C. Delgado Kloos
which a signicant amount of interaction is taking place. The initial conditions
were also to deploy the data gathering in a scalable and non-intrusive way.
The virtual machine was made available at the beginning of the course. Out
of the 248 students that signed out for the course, a total of 220 downloaded
the machine (88:71%). Out of the remaining 28 students (11:29%), most of them
opted to use their own congured environment. The large percentage of students
that decided to use the machine shows its acceptance as the course tool.
The number of downloads, though turned out to be not a good estimation
of the true activity carried out by the students in those machines where the
recording mechanism was not disabled. The events received in the rst half of
the course (in which students worked in all the activities in pairs) were 48; 342
for a total of 115 students (an average of 420 events per student).
3.1 Activity outside the LMS
An important side-eect of placing the data-gathering phase outside of the LMS
and into a fully congured environment was to be able to measure the percentage
of URLs that were related to the LMS. In other words, by exploring the events
encoding a visit to a URL we can have a rst look at the percentage of trac
that goes to other sites.
Out of the almost 49; 000 events, 15; 507 (32:07%) were events in which a
URL was opened with the browser. When counting the number of unique URLs,
this number falls down to 8; 669. Out of these, only 2; 471 (28:51%) pointed to
the LMS. An initial interpretation (pending a more thorough analysis) seems to
suggest that students interact with a large number of resources that are outside
of the LMS.
4 Discussion and Future Work
In this paper a context has been described in which in order to assess the degree
of interaction that students are having with a previously detected set of tools and
among themselves, the LMS oers a very poor coverage. The context is derived
from the adoption of learning outcomes that require procedural activities with
tools and functionality beyond the scope of a conventional LMS>
The described approach proposes extending the scope of the data-gathering
techniques to include a fully congured virtual machine containing all the re-
quired tools as well as a mechanism to record a subset of the most representative
events. A detailed description of the terms of use of the machine with instruc-
tions on how to disable the recording mechanism, as well as how to check, modify
or delete the information, was included with the machine. The machine is also
congured to establish a bidirectional communication channel with a central
server to send the recorded events and receive new resources that are shown in a
desktop widget as actuators on the learning environment. The received data has
been encoded using the CAM format and is being prepared to perform a more

Stepping out of the box. Towards analytics outside the LMS 9
sophisticated algorithm to detect special patterns to detect early which students
are not using properly the given tools.
The work is still in its preliminary stage in the sense that only the data-
gathering stage has been successfully deployed. Still, the approach has been
shown to be scalable (more than 200 students) and in-obtrusive (students do
not sense that the events are being recorded). The obvious line for future work
is to identify those variables of the recorded events are more suitable to make
predictions of those students that are not using properly the tools included in
the machine.
A second line of work has also been conceived to combine the recorded events
and the information extracted from the LMS to detect potential anomalies in
the collaborative part of the course. The challenges in this context are bigger
because teams are suppose to meet regularly on face-to-face meetings in which
there is no type of event recording.
Acknowledgment
Work partially funded by the Learn3 project, \Plan Nacional de I+D+I TIN2008-
05163/TSI", the Best Practice Network ICOPER (Grant No. ECP-2007-EDU-
417007), the Accion Integrada Ref. DE2009-0051, and the \Emadrid: Investi-
gacion y desarrollo de tecnologas para el e-learning en la Comunidad de Madrid"
project (S2009/TIC-1650).
References
1. Criteria for accrediting computing programs. Tech. rep., ABET Accreditation
Board for Engineering and Technology (2007)
2. Anaya, A.R., Boticario, J.G.: Application of machine learning techniques to analyse
student interactions and improve the collaboration process. Expert Systems with
Applications 38(2), 1171{1181 (Feb 2011)
3. Anderson, T.: Getting the Mix Right Again: An updated and theoretical rationale
for interaction Equivalency of Interaction. The International Review of Research
in Open and Distance Learning 4(2) (2003)
4. Arnold, K.: Signals: Applying Academic Analytics. EDUCAUSE Quarterly 33(1),
10 (2010)
5. Campbell, J., DeBlois, P., Oblinger, D.: Academic Analytics: A New Tool for a
New Era. Educause Review 42(4), 40{57 (2007)
6. Chatti, M., Jarke, M., Frosch-Wilke, D.: The future of e-learning: a shift to knowl-
edge networking and social software. International journal of knowledge and learn-
ing 3(4), 404{420 (2007)
7. Dawson, S.: `Seeing' the learning community: An exploration of the development of
a resource for monitoring online student networking. British Journal of Educational
Technology 41(5), 736{752 (Sep 2010)
8. Elias, T.: Learning Analytics : Denitions , Processes and Potential (2011)
9. Gaspar, A., Langevin, S., Armitage, W., Rideout, M.: March of the (virtual) ma-
chines: past, present, and future milestones in the adoption of virtualization in
computing education. Journal of Computing Sciences in Colleges 23(5), 123{132
(2008)

10 A. Pardo, C. Delgado Kloos
10. Goldstein, P., Katz, R.: Academic analytics: the uses of management information
and technology in higher education, chap. 1, pp. 1{12. Educause (2005)
11. Jovanovic, J., Gasevic, D., Brooks, C., Devedzic, V., Hatala, M., Eap, T., Richards,
G.: Using Semantic Web technologies to analyze learning content. IEEE Internet
Computing 11(5), 45{53 (2007)
12. Light, R.: Making the most of college: Students speak their minds. Harvard Univ
Pr (2001)
13. Macfadyen, L.P., Dawson, S.: Mining LMS data to develop an \early warning
system" for educators: A proof of concept. Computers & Education 54(2), 588{
599 (Feb 2010)
14. Mazza, R., Dimitrova, V.: Visualising student tracking data to support instructors
in web-based distance education. In: Proceedings of the 13th international World
Wide Web conference on Alternate track papers & posters. pp. 154{161. ACM
(2004)
15. Mazza, R.: A graphical tool for monitoring the usage of modules in course man-
agement systems. In: 4370, L. (ed.) Visual Information Expert Workshop, VIEW
2006. pp. 164{172. Springer (2006)
16. Mu~noz Merino, P.J., Pardo, A., Kloos, C.D., Mu~noz organero, M., Wolpers, M.,
Niemann, K., Augustin, S.: CAM in the Semantic Web World. In: International
Conference on Semantic Systems. p. Accepted as Poster (2010)
17. Norma, D.A., Spohrer, J.C.: Learner-centered education. Communications of the
ACM 39(4), 24{27 (Jan 1996)
18. Scheel, M., Friedrich, M., Jahn, M., Kirschenmann, U., Niemann, K., Schmitz,
H., Wolpers, M.: Self-monitoring for Computer Users. Informatik (2009)
19. Schmitz, H.C., Wolpers, M., Kirschenmann, U., Niemann, K.: Contextualized At-
tention Metadata, vol. 13, chap. 8. Cambridge University Press (Sep 2007)
20. Schmitz, H., Scheel, M., Friedrich, M., Jahn, M., Niemann, K., Wolpers, M.:
CAMera for PLE. Learning in the Synergy of Multiple Disciplines pp. 507{520
(2009)
21. Waycott, J., Bennett, S., Kennedy, G., Dalgarno, B., Gray, K.: Digital divides? Stu-
dent and sta perceptions of information and communication technologies. Com-
puters & Education 54(4), 1202{1211 (May 2010)
22. Wolpers, M., Najjar, J., Verbert, K., Duval, E.: Tracking actual usage: the attention
metadata approach. Journal of Technology Education & Society 10(3), 106{121
(2007)