Augmenting Human Memory using Personal Lifelogs

Yi Chen
Centre for Digital Video Processing
Dublin City University
Dublin 9, Ireland
ychen@computing.dcu.ie
Gareth J. F. Jones
Centre for Digital Video Processing
Dublin City University
Dublin 9, Ireland
gjones@computing.dcu.ie


ABSTRACT
Memory is a key human facility to support life activities,
including social interactions, life management and problem
solving. Unfortunately, our memory is not perfect. Normal
individuals will have occasional memory problems which can be
frustrating, while those with memory impairments can often
experience a greatly reduced quality of life. Augmenting memory
has the potential to make normal individuals more effective, and
those with significant memory problems to have a higher general
quality of life. Current technologies are now making it possible to
automatically capture and store daily life experiences over an
extended period, potentially even over a lifetime. This type of
data collection, often referred to as a personal life log (PLL), can
include data such as continuously captured pictures or videos
from a first person perspective, scanned copies of archival
material such as books, electronic documents read or created, and
emails and SMS messages sent and received, along with context
data of time of capture and access and location via GPS sensors.
PLLs offer the potential for memory augmentation. Existing work
on PLLs has focused on the technologies of data capture and
retrieval, but little work has been done to explore how these
captured data and retrieval techniques can be applied to actual use
by normal people in supporting their memory. In this paper, we
explore the needs for augmenting human memory from normal
people based on the psychology literature on mechanisms about
memory problems, and discuss the possible functions that PLLs
can provide to support these memory augmentation needs. Based
on this, we also suggest guidelines for data for capture, retrieval
needs and computer-based interface design. Finally we introduce
our work-in-process prototype PLL search system in the iCLIPS
project to give an example of augmenting human memory with
PLLs and computer based interfaces.
Categories and Subject Descriptors
H.3.3 [Information Search and Retrieval]: Search and Retrieval
- Search process, Query formulation, H.5.2 [User Interfaces
(D.2.2, H.1.2, I.3.6)]: Graphical user interfaces (GUI),
Prototyping, User-centered design
General Terms
Algorithms, Design, Human Factors.
Keywords
Augmented Human Memory, Context-Aware Retrieval, Lifelogs,
Personal Information Archives
1. INTRODUCTION
Memory is a key human facility inextricably integrated in our
ability to function as humans. Our functioning as humans is
dependent to a very significant extent on our ability to recall
information relevant to our current context, be it a casual chat
with a friend, remembering where you put something, the time of
the next train or some complex theory you need to solve a
problem in the laboratory. Our effectiveness at performing many
tasks relies on our efficiency and accuracy in reliably recalling
the relevant information. Unfortunately humans are frequently
unable to reliably recall the correct information when needed.
People with significant memory problems (e.g. amnesic patients)
usually face considerable difficulty in functioning as happy
integrated members of society. Other people, although having
much less noticeable memory problems compared with the
amnesic patients, may also experience some degree of difficulties
in learning and retrieving information from their memory for
various reasons. In this paper, we use the phrase normal people to
refer to individuals with normal memory and normal lifestyles, as
opposed to amnesic or mentally impaired patients. The
desirability of a reliable and effective memory means that
augmenting memory is a potentially valuable technology for
many classes of people.
Normal individuals might use a memory augmentation tool to
look up partially remembered details from events from their life
in many private, social or work situations. The augmented
memory application itself might proactively monitor their context
and bring to their attention information from their previous life
experiences which may be of assistance or interest to their current
situation. Details from these experiences could be integrated into
personal narratives for use either in self reflection or to enable
experiences to be shared with friends [1]. Sufficiently powerful
augmented memories could not just support their users, but
actually extend the user’s capabilities to enable them to perform
new tasks or existing tasks more efficiently or faster.
In order to provide augmented memory applications however we
need some means to capture, store and then access personal
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
Augmented Human Conference, April 2–3, 2010, Megève, France.
Copyright 2010 ACM 978-1-60558-825-4/10/04…$10.00.

information from a person’s life experiences to form an
augmented memory. The emerging area of digital life logging is
beginning to provide the technologies needed to support these
applications. Personal lifelogs (PLLs) aim to digitally record and
store many features of an individual’s life experiences. These can
include details of visual and audio experiences, documents
created or read, the user’s location, etc. While lifelog technologies
fall short of genuinely mimicking the complexities and processes
of human memory, they are already offering the promise of life
enhancing human augmentation, especially for episodic memory
impaired patients, that is people who have problems remembering
their daily experiences [2].
Existing studies in lifelogs have concentrated primarily on the
physical capture and storage of data. One of the major activities in
this area which has explored this topic in detail relates to Gordon
Bell’s experiences of digitizing his life described in [3]. Bell
explores the topic of “e-memories” and “Total Recall”
technologies related through this own experiences of digitizing
his life. While his work provides significant insights into the
potential and issues or digital memories, it focuses very much on
the technologies of capture and storage, and potential
applications. Our work in the iCLIPS project in the Centre for
Digital Video Processing (CDVP) at Dublin City University1 is
exploring capture and search of personal information archives
looking not just at data capture, but also the development of
effective content indexing and search, and importantly in relation
to this paper, that we are looking at the processes of human
memory, the form and impact of memory failures and how these
might be overcome using search of PLLs. In our work, we are
concentrating on memory failures typically encountered by
normal people, and using these to guide the development of a
prototype interface to access a PLL as an augmented memory.
The remainder of this paper is organised as follows. In section 2
we examine PLLs and related works on digital memory aid tools
in a little more detail, Section 3 then looks at theoretical models
of memory from the psychology literature, reviews some existing
empirical studies regarding normal people’s memory problems
and memory supports needs in their daily life, and discuss the
possible function that PLLs may be able to provide for
augmenting human memory In section 4 we postulate the
guidelines for developing PLLs systems to augment human
memory, giving suggestions on computer based interface
designing, types of data to be capture, and retrieval techniques
required. And finally section 5 we introduce the iCLIPS project
and our prototype application.
2. BACKGROUND
2.1 Personal Lifelogs (PLLs)
PLLs are typically captured using a range of software and
hardware devices. A separate lifelog is captured for each
individual. In our work we use software applications to log all
activity on the individual’s desktop and laptop computers. This
involves recording files created, edited and opened, logging web
pages accessed, and archiving emails sent and received.
Peripheral devices are used to continuously record other
potentially significant data streams. These include visual
information recorded using a wearable camera, in our case the

1 http://www.cdvp.dcu.ie/iCLIPS
Microsoft SenseCam2, or camcorders, some projects also use
audio recording to record conversations, the most important
source of our conventional daily communication. Due to privacy
concerns related to continuous capture of audio data, we do not
capture audio data in our work. However, other communications
sources such as SMS messages and Twitter feeds can be
monitored and included in a PLL. In addition there is a wide
range of context information that can also be recorded. For
example, location can be monitored using GPS sensors which
then look up named locations in gazetteers, individuals present
can often be inferred by monitoring nearby Bluetooth enabled
devices, date and time are often easily be captured, and are very
powerful context data for search of information. Another
interesting source of data is biometrics. Research has shown a
correlation between measurable biometric responses such as heart
rate and skin conductance and temperature, personal arousal and
memorability of events [4]. Thus capturing these biometric
features can potentially be used to help locate events in a PLL of
potential personal significance to its owner.
2.2 Related Works
There have been a number of studies on developing memory-
supporting applications. But most of current research claiming to
use PLL data in supporting human memory is limited to
presenting streams of captured episodes (e.g. video or audio
records of certain episode) to the user, to have them “re-
experience” the past, to look up required information or “re-
encode” information encountered during that period and
consolidate memory of it (e.g.[2, 5, 6]). While these applications
appear to have promising results in clinical psychology studies
with severe amnesic patients, that is, people who suffer from
serious memory disorders (e.g. [2]), such applications may not be
equally useful for people who have normal memory abilities. For
example, in the study of [2], the subject (patient) can hardly recall
anything that happened to her even after one day’s delay.
Therefore, a simple review and consolidation of past events can
be very helpful for them to maintain necessary episodic memory.
A patient’s lifestyle can be very different from that of normal
working people, in that they have enough time to review their
experiences day by day. Therefore the “rehearsal” type memory
aid (e.g. [5, 6]) is less likely to be favoured by normal people,
unless it contains some important information which is difficult to
remember or if there is some specific information they can’t
recall. For example, it is not unusual that we need to find an
object or a document but don’t remember where it is, or we meet
someone we saw before but can’t recall the person’s name.
Ubiquitous Memories [7] is a tool designed to automatically
retrieve video clips which were captured when an object was
previously presented. The developers also argue it to be a tool to
help people find physical objects. VAM [8] was designed to
automatically retrieve personal information such as the name of a
currently encountered person by automatically detecting the face
of the person. Audio life logs such as iRemember [9] are usually
used to recover information one learned from audio
conversations. Forget-Me-Not [10] helps people find documents
by searching for actions in which the document is involved. The
cues it presents to trigger memories of the target document related
action also include other actions in the day which are presented

2 http://research.microsoft.com/en-us/um/cambridge/projects/sensecam