Visualizing Activities for Self-reflection and
Awareness
Sten Govaerts, Katrien Verbert, Joris Klerkx, and Erik Duval
Katholieke Universiteit Leuven, Department of Computer Science
Celestijnenlaan 200A, B-3001 Heverlee, Belgium
{sten.govaerts,joris.klerkx,katrien.verbert,erik.duval}@cs.kuleuven.be
http://hmdb.cs.kuleuven.be
Abstract. With the large uptake of online learning environments in
educational institutions, students and teachers interact no longer solely
face-to-face and are often geographically dispersed. Distance education
can make it much harder for students to position themselves in the group
of fellow students and to self-reflect on their work. The teacher can have
similar problems by losing the general overview of the class, which makes
it hard to discover potential pitfalls of students in time. To assist with this
problem, we have developed a set of visualizations of learner activities
to increase awareness and to support self-reflection. This paper discusses
those visualizations and evaluates the usability and user satisfaction.
From these studies, first preliminary results are available of the usefulness
of the tool for the students. A planned future evaluation is discussed to
evaluate the real usage and impact of the tool.
Key words: Visualization, Learning activities, Monitoring, Self-reflection,
Usability, User satisfaction, Awareness
1 Introduction
Nowadays, most educational institutions use online learning environments to
support their courses. This evolution enables easy distribution of course mate-
rial, remote participation and collaboration and increased interactivity between
geographically dispersed teachers and students.
On the other hand, some things also substantially change with moving to a
more digital classroom environment. In a real life classroom, students often get
faster feedback on their efforts by the teacher and their peers, and can position
themselves among their fellow students through continuous social contact. In dig-
ital learning environments, it is much harder for students to position themselves.
For example a student does not know if he is working a lot for a course relative
to others or whether he is doing what the teachers expect. The same counts for
online group work, where the level of participation can vary significantly between
different students and some students can dominate others. Different solutions are
available, e.g. careful design of the online course and close follow up of students
[1]. Using self-reflection also increases motivation and participation [2].

2 Visualizing Activities for Self-reflection and Awareness
In this paper, we try to tackle these issues through analyzing and visualizing
learner activities. We believe that visualizations allow users to understand and
discover patterns in data more easily. We also concur with Napoleon Bonaparte’s
famous saying: Un bon croquis vaut mieux qu’un long discours (meaning: A
good sketch is better than a long speech). Visualizations of learner activities can
motivate students, provide feedback on their work [3], reveal participation and
detect who is working harder or taking a free ride. The user activities can also be
interesting for learning material recommendations. Knowing what other students
are studying can be very helpful to make progress with the course.
First, we will look into related research. Then, we will discuss the objectives
of the tool, the design choices and implementation. Afterwards, we will explain
the evaluation. We conclude with possible extensions and future evaluation.
2 Related work
Visualization of personal and social data, such as user activities, social networks
and live logging [4], is a popular research topic in various application domains.
We focus on education, where different approaches have been researched.
Jermann et al. [5] have reviewed several systems that visualize learner activ-
ities to increase awareness and collaboration among learners. The distinction is
made between systems that reflect actions and those that monitor the interac-
tion state. The first category provides basic support for improving awareness of
actions taken on resources, for instance along a timeline. The second category
includes support for aggregating data into a set of high-level indicators that are
displayed to users. For instance, analyzing participation rates with word counts
and message reply delays, and relating this back to an ideal model. Our tool
situates itself in the first and partly in the second category, because it visualizes
student activities over time and also provides some simple statistics on time ex-
penditure and document use. We focus on presenting the data and due to the
difficulty related to it, we are not developing learning models from the data.
Several other systems visualize learner activities with different objectives:
improving collaboration among learners, increasing awareness, supporting self-
reflection and enabling the discovery of peer learners through social network
analysis. CAMera [6] is a tool for personal monitoring and reporting. CAMera
uses Contextualized Attention Metadata (CAM) [7] that capture the interac-
tions of users with resources and tools and provides simple metrics, statistics
and visualizations of the activities of the learner. It also visualizes a social net-
work based on email communication. We build on CAM data, which enables
any system that produces CAM to use the visualization. Colleagues of the de-
velopers found it useful to use, but a larger evaluation is not yet available. The
main difference with CAMera is that our tool uses time expenditure data of stu-
dent groups to analyze higher-level indicators, e.g. CAMera visualizes personal
mouse clicks. The Participation Tool [3] visualizes the online communication of
students doing group work. Janssen et al. showed that students using the Partic-
ipation Tool participated and coordinated more than students without the tool.

Visualizing Activities for Self-reflection and Awareness 3
We focus on general courses and not specifically on group work. This can be
done in the future if we have a use case with group layout data. CourseVis [8]
allows teachers to become aware of social, behavioral and cognitive aspects of
distance learners. Mazza et al. use web log data from online course management
systems to visualize interaction in discussions, quiz performance and page ac-
cesses. Teachers could more quickly grasp the information and regard it as very
useful. Moodog [9] is a Moodle plug-in that visualizes data from the activity logs
to allow students to compare their progress to others and teachers get insight
into the student interactions with the online course. For resources, it shows the
number of unique users, view counts and popularity. For students, it shows total
views, the number of sessions, the total time spent online, the number of viewed
resources and the number of forum threads and posts made. We use similar
metrics as Moodog, but we do not want to commit to one learning platform.
3 Objectives
In the ROLE project1, we research the assembly of learning services, tools and
resources in open personal learning environments. Personal learning environ-
ments (PLE) are user customizable learning environments that support re-use,
creation and mash up of tools, services and resources. PLE’s are often used in
self-regulated learning where self-monitoring and -reflection is important.
Our tool is developed for use in a PLE in the ROLE project and shows an
overview of student activities in a course. Both students and teachers can use
it, because the visualizations can provide self-reflection and awareness of the
on-going activities in a course on both the personal and group level. Other tools
often focus on one of the two [8].
Self-monitoring for learners is important in for example self-regulated
learning and personal learning environments [2]. Having feedback on your work
often leads to improvements. This can be achieved through the teacher, but also
through the analysis of monitoring data. Our visualizations allow learners to see
when, on what and how long they have been working and compare this with the
others. Discovery of averages and trends in the monitoring data is also possible.
Awareness for teachers is also important because they are often left in
the dark on the progress of students until for example an exam. This can be
especially troublesome in online and distant courses. We provide an overview
of student’s time spending and what they have been working on so that the
instructor can analyze this and find patterns to spot problems at an early stage.
For example, students spending too much time on one exercise might indicate
that they have a hard time accomplishing it. The same way teachers can get an
idea of which students need more attention to successfully complete the course.
Time tracking can be helpful for teachers to compare the actual time ex-
penditure against their estimation and sometimes students have to report their
time spending. We provide statistics like the total time spent.
1 The ROLE project, http://www.role-project.eu

4 Visualizing Activities for Self-reflection and Awareness
Learning resource recommendation can be very valuable for both teach-
ers and other learners. Especially in courses where students research topics in-
dependently using a vast collection of data. A teacher might like to know what
is popular and compare this with his preferences. Learners and teachers can dis-
cover new interesting materials to use and mash up in their PLE by knowing
what the others have been studying.
4 Design and Implementation
Now we discuss our design decisions, the visualization choices and the evaluation
data. Finally, we briefly discuss the implementation and current deployments.
4.1 User Interface Design and Visualizations
Fig. 1 shows our tool for usage within personal learning environments (PLE).
Our tool is not developed on top of a specific learning environment to allow the
learner to mash up his own PLE, unlike Moodog [9] and CourseVis [8]. To make
it interoperable with multiple systems, we use CAM [7]. CAM data from another
platform can be easily visualized. The tool will be used in a web-based PLE and
is hence implemented as a web application that can be embedded in a HTML
page or a widget.
Statistics are shown on the right (Box 2) with global course statistics on
time expenditure and document usage. A more graphical line representation
enables comparison. The colored points show simple statistics like minimum,
maximum, average time spent per student and the time spent for the currently
logged in user and for a user selected in one of the visualizations. Each of the
metrics has his own color consistently used in every part of the application.
The recommendation pane at the bottom (Box 3) is a bar that allows
graphical navigation through the document recommendations. The recommen-
dations are based on two simple metrics: the most used documents and the
documents on which the most time was spent. The ROLE project will develop
more advanced recommendations in the future to be plugged in.
The line chart (vis. A in Fig. 1) shows a line for every student in the course.
The horizontal axis shows the dates and the vertical axis displays the total time
spent. The line connects all the timestamps when a student was working. If the
line goes steep up, it means that the student worked rather a lot during that
period. If the line stays flat, the student did not work much on the course. For
example, the student labeled ‘s1’ started only half through the period to put
serious effort in the course, but when he did, he worked very hard for a very
short time. Student ‘s2’ started from the beginning of the course to work slowly
and then worked harder in about the same period as student ‘s1’. This provides
an overview of activities in the course over time and allows students to position
themselves. At the bottom, a smaller version of the visualization is shown with
a slider on top to select a part of the period for analysis of data dense areas.

Visualizing Activities for Self-reflection and Awareness 5
Fig. 1. The user interface with the 3 different visualizations.
The parallel coordinates in visualization B are a common way to display
high-dimensional data [10]. On the vertical axes, we show (i) the total time spent
on the course, (ii) the average time spent on a document, (iii) the number of
documents used and (iv) the average time of the day that the students work. A
student is represented as a polyline connecting the vertices on the vertical axes.
The position of the vertex on the i-th axis corresponds to the i-th data point
coordinate. For example, the green line (the logged in user) works on average
in the early evening and is spending an average time in line with the majority.
He does not use so many different documents and on average looks at these for
a short time. He scores the worst here. The ‘average’ student of the class (in
yellow) is also calculated. This is a much more advanced visualization but can
provide a good overview of the tendencies in the behavior of the students.

6 Visualizing Activities for Self-reflection and Awareness
The bar chart splits the data up in bins for the total time spent and the
number of documents used, and shows the distribution of the students. The
idea originates from Wright [11], who uses the bell curve as a simple way to
evaluate progress in learning. By clicking on the bars you can dig deeper in a
certain segment. The bin of the logged in user is colored green as in the other
visualizations. This visualization can present an overview of the activities in a
course, but also provides a deeper understanding of the details.
4.2 Implementation and deployments
The software is developed in Adobe Flex. Currently the tool is used in four
different settings. For the evaluation, it is used in a computer science course (as
shown in Fig. 1). It is embedded in a PLE prototype as an OpenSocial widget
[12] using CAM data generated by the PLE and will be used in future evaluations
(see section 5.3) in the ROLE project. In our research group it is used as self-
reflection tool to see if it works in this context as well. And it can also connect
to a Moodle database.
4.3 Data for the visualizations
To drive the visualizations, data is needed. In the ROLE project, we use CAM
data generated by the PLE prototype. To capture data for our evaluation with
computer science students, we looked at different time tracking tools. But be-
cause there is no free, cross platform tool available, we decided to create our
own application using Twitter2, a popular micro-blogging service. Many Twit-
ter clients are available and it provides an easy API. Students can their track
time by sending a message to Twitter using hashtags to identify it as time track-
ing message and a description of the task and duration. Because the students
decide themselves what they make public, privacy is not a real problem here.
5 Evaluation
The usability and user satisfaction was evaluated in a real life setting with twelve
computer science students (10 male and 2 female, from 22 to 25 years old) taking
a user interfaces course in April-May 2010. This does not prove that the tool
really assists students and teachers with self-reflection and awareness, but that
they can use and understand the tool, which is a conditio sine qua non. We
plan to evaluate this later (see section 5.3). We now elaborate on the evaluation
techniques, the results and future evaluation plans.
5.1 Evaluation techniques
We use the usability quality components of Jakob Nielsen [13]: learnability (how
easy is it to work with the tool for the first time?), efficiency (how quickly can
2 Twitter, http://twitter.com

Visualizing Activities for Self-reflection and Awareness 7
users perform tasks?), memorability (do users remember how to work with the
tool after a period?), errors (how many errors do users make?) and satisfaction.
We did two evaluation sessions. After a week of time tracking, a task-based
interview was conducted using the think-aloud protocol [14] where students used
the tool for the first time. With think-aloud, users are asked to say whatever
they do, look at, think and so on, while doing a task. After selecting the user
interfaces course, they had to explain what they saw on every visualization and
show where they were displayed. Next, we asked to explain the statistics window.
In this session we evaluated their understanding of the visualizations and thereby
the learnability, errors and efficiency. We did not measure the time to perform a
task, because the focus of the tool is exploration and finding interesting insights.
A month later, we evaluated the user satisfaction. To measure this, we looked
at existing surveys. Tullis et al. [15] conclude after comparing four surveys that
the System Usability Scale (SUS) [16] yields amongst the most reliable results.
Hence, we choose SUS, which has 10 questions using a Likert scale [17]. In
addition, we apply the Microsoft Desirability Toolkit (MSDT) [18]. The user
has to choose all adjectives describing the system out of a list of 118, then to
pick the top 5 adjectives and to discuss these. Through this explanation session,
we can get more detail of the user’s experience and opinion of the software.
The evaluation session was done both in an interview and online, due to time
constraints. First, the SUS survey was filled out. Then, we asked for the positive
and negative experiences with the tool and it concluded with the MSDT.
5.2 Evaluation results
Learnability, errors and efficiency was evaluated with 7 of the 12 partici-
pating students, due to unavailability. Generally, the students could explain well
what they saw on the visualizations. Only two students needed explanation that
every line in the parallel coordinates and line chart represented a student. Two
students had trouble finding data points (the info boxes visible in Fig. 1, box A)
in the line chart, because they are only selectable at the data point itself and
not everywhere on the line. Making a selection of the data in the line chart was
not clear for 6 out of 7 students. Once told, they found it obvious. Four students
would like rather less detailed measures on the axes on the line and bar charts,
e.g. 5h instead of 4:56:32. Names on every line in the line chart and parallel coor-
dinates was requested by two, which will likely generate visual clutter for large
groups. No problems with the statistics window were discovered. Some other
small usability remarks were made on the font sizes and missing axes’ titles.
In conclusion, the visualizations themselves were clear to all students. The
operation of the line chart’s selection tool needs documentation using e.g. a
tooltip. The learnability aspect was good. We did not time the efficiency, though
overall no real problems were uncovered, but improving the selection of the lines
could improve this. Some small errors were discovered during the evaluation, but
nothing that makes recovery necessary. Bugs and some small usability remarks
were fixed before we released the tool to the students.

8 Visualizing Activities for Self-reflection and Awareness
Fig. 2. A word cloud based on the frequency of the selected product reaction cards.
The User Satisfaction was measured with nine students, of which five were
veterans of the first round. The average SUS score was 73,33% with a standard
deviation of 9,35. Three students were above 80% and seven above 70%. The
two students scoring lower than 70% did not participate in the first round.
From the MSDT results we created the word cloud in Fig. 2 based on the fre-
quency of all the selected words. In Fig. 2 we clearly see that ‘Usable’, ‘Useful’,
‘Understandable’ and ‘Organized’ are the most selected words. This indicates
that the students have some use for the tool. Which is amplified by ‘Relevant’,
‘Effective’ and ‘Meaningful’. In the future, we will investigate this further. Words
like ‘Straightforward’, ‘Easy to use’ and ‘Intuitive’ indicate a high level of us-
ability, comparable to the SUS results. Three students selected ‘Complex’ and
two of these students stated that this was only their initial reaction to the tool.
The other student saw it as a positive thing, because for him the complex design
meant it was advanced or evolved and not difficult to use.
Verbal protocol analysis is applied to the top five choices of the students. To
do this, we count the negative and the positive words. 42 words were positive
(93%) and 3 words were negative (7%). The negative words were ‘overwhelming’
and ‘cluttered’. The three students said that this was their initial reaction, but
that the problem disappeared once they understood the visualizations. Both
results indicate a quite high user satisfaction, but it can be improved by providing
functionality to inform first time users on how to use the visualizations.
Next we asked them why the tool is useful for them. Six used it to see
how much the others worked on the course for comparison. Three were mainly
interested in their own data for reporting. This result is very much in line with
our potential use cases. As for their favorite visualization: four students voted
for the line chart, three for the parallel coordinates and two for the bar chart.
Initially, we included three visualizations because we did not know what people

Visualizing Activities for Self-reflection and Awareness 9
would like to use. We were a bit surprised that the parallel coordinates scored so
high, because it is harder to grasp at first sight. The students, who voted for this,
said they liked it for the clear overview and to see tendencies more easily. Since
the choice of the visualizations is personal, we could make this configurable.
5.3 Future Evaluation
Our evaluation only scratched the surface of measuring the real use and impact
of the tool. This triggers questions like: does it change learning behavior and time
expenditure of students? Does the tool improve in their planning and results?
Further evaluation of the preliminary positive results on the usefulness of
the tool from the MSDT is planned in the coming months. We will use a large
engineering course (393 students) at the RWTH-Aachen University. Our tool will
be embedded in their learning environment and their actions on the system are
currently tracked. We plan to measure and evaluate at 3 points in time: about
a month before the examinations, right before the exams and after. We will
analyze the real usage data of the tool and measure the usefulness and impact
through surveys. Thanks to the large amount of students, we hope to create a
study with statistically relevant outcome. To evaluate if the tool would be useful
for teachers, we plan to setup an evaluation with teachers of the agricultural
research group, CGIAR3, using Moodle.
6 Conclusion and Future Work
In this paper we presented three visualizations to get an overview and explore the
activities in a classroom and find new patterns, used for self-monitoring for stu-
dents and to improve teachers’ awareness. The tool is deployed in three different
settings. Through a usability test with interviews and the think aloud protocol,
we found out that the learnability is high. The number of errors is low and no
unrecoverable errors are encountered. We use the SUS user satisfaction test and
achieved an average score of 73%. The results from the MSDT strengthened this.
Our tool is currently implemented in a PLE prototype of the ROLE project.
This will be used in different test beds, from which we hope to gather more
feedback and usage statistics. One of the major tasks will be expanding the
evaluation as described in section 5.3 to investigate whether the tool really helps
with self-reflection and assists teachers. There are also plans to deploy the visu-
alizations on the ZOEP4 video portal for teachers. We will also look into other
visualizations and data source possibilities to extend the tool’s functionality.
Acknowledgments. The research leading to these results has received fund-
ing from the European Community Seventh Framework Programme (FP7/2007-
3 Consultative Group on International Agricultural Research, http://www.cgiar.
org/
4 The ZOEP video portal, http://zoep.windesheim.nl/

10 Visualizing Activities for Self-reflection and Awareness
2013) under grant agreement no 231396 (ROLE). Katrien Verbert is a Postdoc-
toral Fellow of the Research Foundation - Flanders (FWO).
References
1. Smith-Gratto, K.: Report to the distance education evaluation task force distance
education best practices and problems North Carolina A & T State University. In:
http://qed.ncat.edu/ir&p/report.htm. (November 1999)
2. Zimmerman, B.J.: Becoming a self-regulated learner: An overview. Theory Into
Practice 41(2) (2002) 65–70
3. Janssen, J., Erkens, G., Kanselaar, G., Jaspers, J.G.M.: Visualization of participa-
tion: Does it contribute to successful computer-supported collaborative learning?
Computers & Education 49(4) (2007) 1037–1065
4. Bell, G., Gemmell, J.: Total Recall: How the E-Memory Revolution Will Change
Everything. Dutton Adult (2009)
5. Soller, A., Mart´ınez-Mone´s, A., Jermann, P., Muehlenbrock, M.: From mirroring
to guiding: A review of state of the art technology for supporting collaborative
learning. I. J. Artificial Intelligence in Education 15(4) (2005) 261–290
6. Schmitz, H.C., Scheffel, M., Friedrich, M., Jahn, M., Niemann, K., Wolpers, M.:
Camera for PLE. In: EC-TEL. Volume 5794 of Lecture Notes in Computer Science.,
Springer (2009) 507–520
7. Wolpers, M., Najjar, J., Verbert, K., Duval, E.: Tracking actual usage: the attention
metadata approach. Educational Technology & Society 10(3) (2007) 106–121
8. Mazza, R., Dimitrova, V.: Visualising student tracking data to support instruc-
tors in web-based distance education. In: Proceedings of the 13th international
conference on World Wide Web, WWW, ACM (2004) 154–161
9. Zhang, H., Almeroth, K., Knight, A., Bulger, M., Mayer, R.: Moodog: Track-
ing students’ online learning activities. In: Proceedings of World Conference on
Educational Multimedia, Hypermedia and Telecommunications 2007, Vancouver,
Canada, AACE (2007) 4415–4422
10. Inselberg, A.: Parallel coordinates. In Liu, L., O¨zsu, M.T., eds.: Encyclopedia of
Database Systems. Springer US (2009) 2018–2024
11. Peter W. D. Wright, P.D.W.: Tests and measurements for the parent, teacher,
advocate & attorney
12. Mitchell-Wong, J., Kowalczyk, R., Roshelova, A., Joy, B., Tsai, H.: Opensocial:
From social networks to social ecosystem. In: Digital EcoSystems and Technologies
Conference, 2007. DEST ’07. Inaugural IEEE-IES. (21-23 2007) 361 –366
13. Nielsen, J.: Usability Engineering. 1st edn. Morgan Kaufmann (1993)
14. van Someren, M., Barnard, Y.F., Sandberg, J.A.: The think aloud method: a
practical guide to modelling cognitive processes. Academic Press, London (1994)
15. Tullis, T.S., Stetson, J.N.: A comparison of questionnaires for assessing website
usability. In: Usability Professionals’ Association Conference, UPA. (2004)
16. Brooke, J.: SUS: A quick and dirty usability scale. In: Usability Evaluation in
Industry. Taylor & Francis., London (1996)
17. Likert, R.: A technique for the measurement of attitudes. Archives of Psychology
22(140) (1932) 1–55
18. Benedek, J., Miner, T.: Measuring desirability: New methods for evaluating desir-
ability in a usability lab setting. In: Proceedings of UPA Conference. (2002)