Semantically Reconnecting Fragmented Information through User Activity Monitoring
Abstract
Today information items on users workstations are usually stored in separate collections depending on their format. This results in a disconnect between information systems and user needs leading to high lookup times during task re- lated information retrieval. This paper presents an approach to reduce document based information fragmentation by se- mantically reconnecting electronic documents to each other without imposing additional training or tagging workload on the user. To this end the actions knowledge workers per- form on their desktop are transparently monitored to ana- lyze the users interaction with his computer system. These action metadata are further clustered by superordinate ac- tivities performed by the user. Finally documents attached to window instances within the identified activity clusters are semantically related to each other reducing the fragmen- tation of their contained information. This allows a subse- quent associative information discovery navigating from one document instance to other related document instances. A prototypical implementation and evaluation in a small scale testing setup indicates the validity of the approach.
Semantically Reconnecting Fragmented Information through User Activity Monitoring
through User Activity Monitoring
Hinnerk Brügmann
Information Systems II
University of
Erlangen-Nuremberg
hinnerk.bruegmann@
wiso.uni-erlangen.de
Matthias Kurz
Information Systems II
University of
Erlangen-Nuremberg
matthias.kurz@
wiso.uni-erlangen.de
Dieter Sauer
DATEV eG
Nuremberg, Germany
dieter.sauer1@
t-online.de
ABSTRACT
Today information items on user’s workstations are usually
stored in separate collections depending on their format.
This results in a disconnect between information systems
and user needs leading to high lookup times during task re-
lated information retrieval. This paper presents an approach
to reduce document based information fragmentation by se-
mantically reconnecting electronic documents to each other
without imposing additional training or tagging workload
on the user. To this end the actions knowledge workers per-
form on their desktop are transparently monitored to ana-
lyze the user’s interaction with his computer system. These
action metadata are further clustered by superordinate ac-
tivities performed by the user. Finally documents attached
to window instances within the identified activity clusters
are semantically related to each other reducing the fragmen-
tation of their contained information. This allows a subse-
quent associative information discovery navigating from one
document instance to other related document instances. A
prototypical implementation and evaluation in a small scale
testing setup indicates the validity of the approach.
1. INTRODUCTION
Due to the high importance computer-mediated communi-
cation and IT systems have in the execution of today’s busi-
ness processes, enterprises are a main driver in the over-
all growth of information by generating a rapidly increasing
amount of information as by-product of their business ac-
tivities [38]. Lyman and Varian estimate the annual growth
of newly created information in enterprises to be 30% [25].
According to studies of Merrill Lynch and Ferris Research
only 20% - 40% of this information is stored in a structured,
semantically described form in electronic databases or struc-
tured business applications (e.g. SAP R/3) [13, 5]. The
other 60% - 80% are contained in the unstructured form of
10th International Conference on Wirtschaftsinformatik,
16th − 18th February 2011, Zurich, Switzerland
electronic documents1 (e.g., Microsoft Office files, e-mails,
images, multimedia files, or web based content). This is
even more dramatic as the annual growth rate of newly cre-
ated unstructured content is considerably higher than that
of structured information.
Historically unstructured information was and in most sce-
narios still is managed in an application specific way. Due
to different business requirements, enterprises employ task-
specific applications which leads to the storage and process-
ing of inherently similar information in different places in
the enterprise information architecture. A single task or
workflow will therefore require the use of multiple appli-
cations resulting in different interaction techniques and in-
formation representations to support the user [27]. Tun-
gare et al. labeled this situation Information Fragmentation
where a user’s data are tied to different formats, distributed
across multiple locations, manipulated by different applica-
tions and reside in a generally disconnected manner [34].
Most often information items are stored in separate collec-
tions depending on their formats: documents are saved in
a documents’ folder hierarchy (e.g. in the My Documents
folder), e-mails in a separate mailbox hierarchy, and book-
marks to favorite web sites in another browser hierarchy [19].
This hierarchy separation has several negative outcomes: It
leads to potential redundancy and an increased required ef-
fort to locate document based knowledge. In addition to
being time consuming, managing three (or more) different
hierarchies generates cognitive load when trying to main-
tain a certain degree of consistency between the hierarchies
and in using multiple different applications with inconsistent
interaction designs [20]. Since users associate information
objects with their projects and tasks rather than document
formats [4] this represents a potential disconnect between
information systems and a user’s needs.
Such problems multiply in a multi user enterprise envi-
ronment with numerous competing repositories, document
management systems, shared network drives, and local file
systems for each distinctive user. On one hand a per-
sons ability to overlook the available number of information
sources diminishes as the amount of documents in numer-
ous different repositories grows. On the other hand a multi
user environment with local file system repositories lacks the
1In the following the term document is understood as relat-
ing to electronic files containing unstructured information.
a result individual knowledge workers in such multi user en-
terprise environments spend an even greater part of their
working hours looking for the correct and most up-to-date
information needed in workflow steps or tasks.
The research contribution presented in this paper is a method
to reduce document based information fragmentation by se-
mantically reconnecting electronic documents to each other
without imposing additional training or tagging workload
on the user. To this end the actions knowledge workers per-
form on their desktop are transparently monitored to ana-
lyze the user’s interaction with his computer system. These
action metadata are further clustered by superordinate ac-
tivities performed by the user. Actions are clustered if their
associated desktop operations can be related to each other
by combining multiple relationship building algorithms. Fi-
nally documents attached to window instances within the
identified activity clusters are semantically related to each
other reducing the fragmentation of their contained infor-
mation.
The paper is organized as follows: Section 2 presents a re-
view of current research. This is followed by the description
of the presented approach in section 3 consisting of the cap-
turing of semantic user activity context, the subsequent clus-
tering of these activity contexts, and the final inference of
semantic interrelations between documents. Section 4 shows
the results of evaluating a prototypical implementation of
the approach in a small scale testing setup. The paper is
concluded with a summary and an outlook to ongoing re-
search and open questions.
2. RELATED WORK
Previous research approaches to establish and utilize re-
lationships between documents can be separated into two
groups. The first group makes use of the content of docu-
ments to detect similarity or references in documents, while
the second group of work relies on analyzing user activity
involving documents usage to infer interrelations.
While some techniques applied by the first group of ap-
proaches (e.g. Named Entity Recognition) are already quite
advanced, more sophisticated parsing of document content
(e.g. Natural Language Processing) is still problematic due
to high manual training requirements of the applied algo-
rithms [8]. Additionally content based approaches are often
limited to text documents (e.g., multimedia files or files in
proprietary formats) [6].
The second group of approaches focuses on the activities of
a user around individual documents. Relying on tasks as the
principle means of document relationship discovery requires
identifying the connections of one piece of information in one
application to task-related information in the same or other
applications. The project Stuff I’ve Seen [12] focuses on doc-
ument metadata and in part document content to create a
context-snapshot of the time when a document was accessed.
A user might then at a later time query the Stuff I’ve Seen
datastore with arbitrary keywords to be presented with a list
of documents he accessed earlier. The research prototype
Phlat [9] combines keyword and property-value search with
faceted browsing functionality. The tagging of documents
which serves as the basis for Phlat ’s superimposed search
UI has to be manually performed by the user though. Both
Stuff I’ve Seen as well as Phlat aim at enhancing informa-
tion retrieval but do not reconnect previously fragmented
information contained in different documents. The task-
centered tools UMEA [21], TaskMaster [2] and TaskTracer
[11, 33] address this to some degree by capturing the desktop
context of project or task related user actions. All three ap-
proaches require the user to manually enter the specific task
or project on which he is working at any current time. This
necessity is avoided by the IBM prototype Activity Explorer
which makes use of document-centric user activities within
a collaborative document management system to add rela-
tions between document instances [14]. This approach relies
on a specifically customized DMS environment though and
does not detect user activity outside this proprietary work
context.
The research projects SWISH [28], Smart Desktop [24], Dy-
onipos [17], APOSDLE [23], and UICO [31] are all en-
gaged in automated task prediction and task switch detec-
tion within personal desktop environments. The first four
approaches rely on textual metadata of surveyed desktop
resources to distinguish task contexts. On top of that the
UICO prototype also takes chronological metadata of con-
textual activities into account. While all five prototypes
show high detection accuracy this comes at the cost of ini-
tially training the employed machine learning mechanisms to
specific application domains [30]. Besides this initial train-
ing UICO requires an additional step to model the ontology
classes.
A limiting factor to the adoption of Knowledge Management
Systems (KMS) is often seen in lacking user acceptance [26].
Steve Bailey (Senior Adviser for Records Management is-
sues at JISC infoNet) summarizes the issues employees have
with the overhead generated by IT systems to improve the
management of unstructured content [1]: “As far as the av-
erage user is concerned, the EDRMS [(Electronic Document
and Records Management System)] is something they didn’t
want, don’t like and can’t use. As such, its no wonder that
so few users accept them. As one person once said to me
’making me use an EDRMS is like asking a plasterer to use
a hammer!” This low user acceptance results mostly from
knowledge workers feeling burdened by additionally work-
load caused by e.g., manual tagging or categorization in
KMS. Incidentally the described approaches to the manage-
ment of unstructured information require some initial knowl-
edge investment before a critical mass is reached which can
be used to endow user benefit (e.g. a DMS only showing rele-
vant search results after a certain amount of manual tagging
on contained documents has been performed). The addi-
tional effort is felt by the employees to be distracting from
the subjectively more important real work and is therefore
often circumvented or altogether ignored [15].
3. CONSENSE APPROACH
The presented approach is set within the encompassing
research context of the ConSense project2 which aims to
gather document related metadata within enterprise envi-
ronments to deduct semantic relations between documents
2www.consense-project.com
sons, projects or products) to allow a semantic exploration
of the enterprise information landscape [6]. The presented
approach forms one of several pillars to capture semantic
metadata on client workstations. While reasoning within
the semantic network is a crucial part of the encompassing
project it occurs at a later stage after sensor data has been
gathered and aggregated3 and is therefore not in the focus
of this paper.
3.1 Semantic Activity Context
According to Kuutti, user activities can be considered as
having three hierarchical levels: activity, action, and oper-
ation, which correspond to motive, goal, and conditions of
the task for which the activity is performed [22]. An activ-
ity may be achieved through a variety of actions. Similarly,
operations may contribute to a variety of actions. In the
following a task is regarded as containing one or more dis-
tinctive activities which in turn consists of various actions
which all relate to the same object or motive (see figure 1).
Actions in turn consist of any number of (desktop) opera-
tions with an action always having a particular goal within
the context of the task-induced activity motive [35]. As will
be shown for the purpose of the presented approach it is suf-
ficient to focus on the lower three levels of activity, action,
and operation to establish relations between documents con-
taining fragmented information. The encompassing level of
tasks spanning potentially multiple days is therefore not in
further scope.
Figure 1: Tasks, activities, actions, and operations
[22]
When evaluating user desktop activity each corresponding
action has a context consisting of the sum of all other ac-
tions taken by a user as well as the respective workplace en-
vironment during the user action. The individual elements
comprising such context can be separated into two dimen-
sions: time and scope. The dimension of time is split into
context elements which happened before, during, and after
the user action. The dimension of scope differentiates among
user actions targeting a specific document and the general
desktop environment visible to the user. This desktop envi-
ronment in the context of a document access consists of all
opened and visible desktop and web applications as well as
all content and documents contained in these applications.
To give a simple business scenario example: A user in the
3See [6] on the subjects of reasoning as well as dealing with
computational complexity in the ConSense project.
Figure 2: Exemplary context of electronic document
access
sales department copies some textual content from docu-
ment price-list.doc into a new document salesoffer.doc
and saves it in the local file system folder customer-alpha.
Figure 2 shows the context of this action. The user op-
eration (in this case copying) allows the assumption (not
certain knowledge) of an existing semantic relation between
the documents price-list.doc and sales-offer.doc. The
metadata gained from observing user operation UO1 might
further be influenced by the corresponding workplace envi-
ronment WE1. So could for example a window positioning
showing both documents next to each other without over-
lap strengthen the assumption of an existing relationship.
Part of the action context is the second user operation UO2.
This in turn can be used to establish a new (or strengthen
an existing) semantic relation of sales-offer.doc and (to a
lesser extend) price-list.doc to other documents contained
in folder customer-alpha. From a modeling perspective
UO2 has its own context, which would then contain UO1 as
an operation that happened shortly before UO2.
3.2 Capturing Semantic Activity
Context
For the purpose of capturing the semantic context of user ac-
tions a client-side sensor plugin is installed on a user’s work-
station. The sensor takes a snapshot of window metadata
whenever changes are occurring within the workspace envi-
ronment either manually triggered by the user (e.g., opening
or moving window instances) or automatically triggered by
the system (e.g., a notification window being shown). Events
being considered changes to the workplace environment are
changes to a window’s X/Y/Z position as well as to its size.
Additional snapshots are taken upon user operations trig-
gering new documents being opened within these window
instances. Table 1 lists the recorded metadata for each win-
dow during a monitoring snapshot.
With multi monitor setups becoming more common [10] the
sensor detects and logs changes in the workstation’s moni-
tor configuration including the monitor’s resolution, physical
position in relation to other monitors and the start and end
time of this specific setup. This allows the later calculation
of the available screen estate to the user for any given time.
In order to analyze these raw action metadata effectively
they are transformed into a hierarchical, chronologically or-
Data type Metadata
int X position of the top left corner
int Y position of the top left corner
int Z axis position
int Monitor id on which the top left cor-
ner is positioned
int Height
int Width
string Unique window identifier
string Unique parent window identifier
string Window title
bool Indicator if the window has UI focus
dered, data structure as depicted in figure 3. The main ob-
ject Working Item contains all workspace related metadata
between the start and end of a user session as indicated by
the operation system’s session start and session end events4.
Within a Working Item window metadata for each snapshot
are contained within a Time Span describing periods during
which no changes to the workplace environment occurred.
These Time Spans contain the window metadata of all win-
dows visible to the user during the duration of the Time
Span. Visibility in this regard is determined by calculating
the relative square size to which a window is visible to the
user using the window’s X/Y/Z position and size metadata.
Due to study results indicating a user preference to perform
a high percentage of window switching on one primary mon-
itor [18] windows being displayed on a secondary monitor as
indicated by the operating system have their relative visibil-
ity reduced by 50%. For windows stretching across monitor
boundaries this reduction is applied according to their par-
tial visibility on secondary monitors. Only metadata for
windows having a computed relative visibility higher than
0.25 are included in the Time Stamp recordings.
The raw window metadata captured in this way offer only
limited leverage to identify substantive relationships be-
tween fragmented document based information. For this
reason it is enriched by metadata relating to the respec-
tive application responsible for the window’s creation as well
as relating to document(s) opened within the window in-
stances. Existing studies regarding the usage of document
and application types during typical task execution patterns
[12, 9] show e-mail as the most common type opened (∼75%),
followed by web pages (∼15%), Microsoft Word files (∼6%),
Microsoft Powerpoint files (∼3%), and various media types
including pictures and audio files (∼4%). Over 6% of all
items were e-mail attachments. Accordingly the sensor plu-
gin’s metadata extraction routines focus on the Microsoft
Office Suite as well as the major web browser families Mi-
crosoft Internet Explorer and Mozilla Firefox. In particular
the Microsoft Outlook element types e-mail, calendar entry,
4The logical session start and session end events are com-
prised of the physical start/stop, sleep/hibernate/stand
by/wake up, user login/logout and remote connection ini-
tiated/terminated events.
5According to [18] windows hidden by more than ∼75% do
not impart any informational value to the user and are there-
fore treated as having no direct relation to the immediate
action context.
Working Item (08:00 – 16:00)with monitor configuration 1024 x 768 and 1280 x 800
Time Span 1visible windows 8:00 – 8:01
Metadata window A
Metadata window B
. . .
Time Span 2visible windows 08:01 – 08:03
Metadata window A
Metadata window C
. . .
. . .
Figure 3: Structure of captured workspace metadata
contact item, and files attached to these elements are in-
cluded in the sensor range. Table 2 lists the (upon availabil-
ity) recorded additional application and document metadata
for each window during a monitoring snapshot.
Table 2: Extracted extended window metadata
Data type Metadata
string Unique id of the application responsi-
ble for the creation of the window
string Name of the application responsible
for the creation of the window
string System path of the application re-
sponsible for the creation of the win-
dow
string[] System path(s) of opened file based
document(s)
string[] Unique id(s) of opened Microsoft Out-
look element(s)
string[] Web page URI(s) opened in web
browser tab(s)
Three categories of documents are treated in more depth
using different content extraction approaches. These cat-
egories are web based HTML documents, elements within
Microsoft Office Outlook, and lastly all other documents.
In the case of web content it is first checked whether the
target is the web address of a HTML file. If this is the case
the sensor accesses the web browser tab’s HTML source and
temporarily stores it on the local machine. In the case of a
Microsoft Outlook element the unique Outlook id initially
extracted is used to capture further metadata for these ele-
ments. Additionally the respective Outlook element is eval-
uated regarding attached files. If found these are recursively
processed in the same way as the originally identified docu-
ments. Lastly for the content extraction from other files the
indexing service inherently contained within the Microsoft
Windows operation system is used to access and extract doc-
ument metadata. If the file system in question is of type New
Technology File System (NTFS), which is commonplace in
modern Microsoft Windows operating systems, additional
metadata from the file system are available for extraction
Because the naming conventions for different types of meta-
data attached to files vary depending on the respective appli-
cation responsible for the creation of the specific file types,
a common semantic as well as syntactic metadata descrip-
tion is necessary. This underlying metadata description is
provided by the ConSense ontology store6 containing both
custom ontology descriptions in the Web Ontology Language
(OWL) format as well as a subset of the Friend-of-a-Friend
(FOAF) and Dublin Core ontologies. In the case of Dublin
Core the original RDF Schema notation has been trans-
formed into OWL mapping rdf:Property predicate types to
owl:AnnotationProperty types. This allows the resulting
Dublin Core OWL to retain an expressivity of OWL-DL
compared to the computationally more complex OWL-Full
superset. The combined expressivity of these ontologies
stays within OWL-DL corresponding to SHOIN (D) using
description logic naming conventions.
This ontology set forms the ground for a static mapping of
raw file attributes to the gathered (RDF encoded) seman-
tic metadata. Additionally the ontology set accessed by the
sensor plugin could be further extended to account for e.g.,
application types beyond the abovely mentioned Microsoft
Office suite. The sensor plugin validates gathered context
input against the common ontologies (without resorting to
expensive inference reasoning) and persists it in the form of
semantic networks in a local Resource Description Frame-
work (RDF) triple-store.
3.3 Clustering Semantic Activity
Context
Utilizing the semantic metadata described in section 3.2 to
relate fragmented document based information to each other
is limited by the underlying assumption that the whole cap-
tured context belongs to a single user activity. Due to multi-
tasking behavior being predominant within all kinds of office
environments and scenarios though this does not hold true
for most real world applications. Knowledge workers spend
a great deal of time engaged in multiple tasks at the same
time [32]. So do for example employees of an information-
technology company spent an average of only 3 minutes per
task before switching to another task [16]. As Salvucci et
al. point out there exist multiple types of multitasking. On
one hand there is concurrent multitasking where a person
changes between different tasks back and forth several times
a second giving the impression that they are performing the
different tasks simultaneously. On the other hand there ex-
ists sequential multitasking where a task is carried out over
minutes or hours until a secondary task interrupting the first
(e.g., a person reading a book while cooking, briefly stirring
the sauce, and then reading again for several minutes in the
book). The basic procedure of human multitasking is al-
ways the same regardless of the kind of multitasking though
as shown in figure 4. An important point in changing from
one task to the next is the time delay which can take quite
long depending on the complexity of the task. In the context
of the presented approach a special subset of multitasking
is of particular interest, namely intermittent processing [3].
Within this scenario several tasks are executed on a com-
6Hosted at http://ontology.consense-project.com
puter system in parallel though the user can put his atten-
tion only on one activity at any time resulting in a de facto
sequential multitasking pattern.
Accordingly the sensor plugin unravels the previously cap-
tured semantic window action metadata and clusters them
according to their superordinate activities. To this end mul-
tiple algorithms are employed to establish semantic relation-
ships between the observed window instances. Initially, the
simplest type of relationship is considered, namely whether
a window instance can be identified as existing across mul-
tiple Time Spans using the unique window identifier as well
as positioning metadata. The second scenario being resolved
by the sensor plugin is the existence of multiple window in-
stances visible within the same Time Spans belonging to the
same application. Such situations often occur when dialog
boxes (e.g. the application asking for confirmation of a spe-
cific user action) are displayed by an application. This is
confirmed if the captured metadata show one of the window
instances containing as parent id the unique handle of the
other. In both scenarios a semantic statement of predicate
belongsToSameActivity interrelating the window instances is
added to the semantic network.
In the next step relationships between windows of different
applications are uncovered which indicate user operations
within the same action and therefore the same superordinate
activity. This is initially done for all windows within a Time
Span using the extracted textual metadata both from the
window instances as well as from their associated documents
(opened and attached). As the starting point for such a
comparison the title metadata of windows and documents is
utilized which is then enriched by the textual metadata of
contained or attached documents.
When employing a basic string comparison algorithm (e.g.
Jaro-Winkler distance [37]) different spelling of keywords
or the presence of textual clutter can taint the relevance
of resulting similarity indices as illustrated in the follow-
ing two exemplary window titles (1) The term Business
Activity Monitoring (BAM) describes a collection of anal-
yses and presentations - Windows Internet Explorer and (2)
C:/Interest/Analyses/BAM - Windows Explorer.
In order to increase the relevance of further processing re-
sults, the sensor plugin sanitizes the textual metadata. In
a first step the application name is stripped from the win-
dow title if it is contained7. In the case of more extensive
textual metadata as shown in (1) the removal of the appli-
cation name alone is not sufficient. As the string contains
a full English sentence it comes with the usual language
specific expletives. To this end the sensor plugin normal-
izes spelling variations using the Porter Stemming algorithm
[36] and separates compound strings into a list of individ-
ual words. Subsequently further splitting is performed on
dashes to separate system paths into single directory name
parts to account for cases as illustrated in example (2). The
resulting list of normalized strings is then cleaned off lan-
guage specific stop words. As mentioned in section 1 the
presented approach is set to avoid task interruption to the
knowledge worker making a manual selection of language
7Within the Microsoft Windows operation system each ap-
plication registers its own name within the registry hive.
contexts suboptimal. To this end the sensor plugin employs
a generic n-gram comparison which utilizes word frequency
classes of the textual metadata to determine the language of
a given (partial) text. According to the detected language,
stop words are then removed from the word list using pro-
vided static stop word lists for the languages Dutch, English,
French, Spanish, Italian, and German.
Finally the word lists for two window instances are com-
pared using a Weighted Tag Similarity algorithm [7] result-
ing in a numeric value in the interval [0,1] indicating the
assumed similarity of the two windows. To reduce the nec-
essary amount of background processing window instances
are only compared if their respective Time Spans lie within
10 minutes of each other.
Any belongsToSameActivity relation concluded by this sim-
ilarity value can only be treated as an assumption with a
certain amount of unreliability. To this end a semantic reifi-
cation statement is added to the belongsToSameActivity re-
lation indicating its reliability as calculated from the simi-
larity value using a fuzzifying function (see figure 5):
rel =
{
2sim2 sim < 0.5
4sim(1− sim2 )− 1 sim ≥ 0.5
Figure 5: Fuzzification of textual similarity to relia-
bility of statement
Figure 6 shows an exemplary semantic network of activity
cluster relations being generated by evaluating window in-
stances across two adjacent Time Spans. Lastly the previ-
ously generated relationships between window instances for
all Time Spans within the current Working Item are con-
densed into a single clustering graph as depicted in figure 7
forming the final activity clusters. The reified reliability is
shown on the graph edges.
3.4 Relating Documents based on
Activity Clusters
Based on the identified activity clusters the sensor plugin re-
lates documents associated to window instances within the
same cluster. To this end a weighted degree centrality C for
each window instance w is calculated8 with W being all win-
dow instances, and rel (w,w′) being the attached reliability
of the belongsToSameActivity relation between windows w
and w′:
W ′w =
{
w′ ∈ W : ∃rel
(
w,w′
)}
Cw =
∑
w′∈W (rel (w,w
′))
|W| − 1
∣
∣W ′w
∣
∣
For every two window instances p and q within the activity
cluster a semantic statement with predicate hasRelationTo
is inserted into the semantic network. Additionally a reifica-
tion statement signifying the reliability of the hasRelationTo
statement with value C(p) ∗ C(q) is added.
Finally in addition to the reliability further reification state-
ments are added containing provenance information to the
statements relating documents to each other. These consist
of the current time stamp, the semantic metadata which
were input for the relation inference, and the identifier for
the presented approach designating it as the source of the
inferred relationship.
As the ConSense research project underlying the presented
approach [6] is focused on multi user enterprise environ-
ments semantic document-relating metadata with relevance
beyond that of the local user (that is higher-level metadata
relating to domain-specific business entities) is then submit-
ted to an enterprise-wide central semantic triple store. This
central metadata repository, forming a virtualized view of
assumptions on real-world relations among business entities
and documents, can in turn be queried by a user or knowl-
edge manager to visualize and cluster relationship types ac-
cording to his task-specific discovery needs. So can for ex-
ample an enterprise legal department query for documents
having a relation to a product line being the subject in a
legal low-product-quality complaint by a client. The query
parameters could then be narrowed to documents having ad-
ditional relations to the internal Quality Management pro-
cess. Alternatively a rule or heuristic based electronic ser-
vice can directly access the semantic aggregation layer and
8Relations (edges) between window instances are treated as
bidirectional for this purpose.
Figure 7: Example of a resulting activity cluster
query or manipulate the semantic network.
4. EVALUATION
To test the viability of the presented approach the prototypi-
cally implementated sensor plugin was installed on the client
desktops of four knowledge workers. During the evaluation
15 Working Items with durations ranging from 5 minutes to
5 hours were generated containing activity patterns of singu-
lar desktop activity, sequential desktop activity and multi-
tasking activity as described in section 3.3. The goal of these
experiments was mainly to to obtain information on the
accuracy of the clustering algorithm both regarding false-
positives (actions included in the clusters) as well as false-
negatives (actual window activity relationships not detected
by the sensor plugin). The test subjects were later asked
to confirm or deny each of the individual belongsToSameAc-
tivity relationships concluded by the sensor plugin, denials
resulting in a false-positive result for that match. In a second
step the subjects were presented with the unconnected win-
dow instances and asked to cluster them according to their
own remembrance of the performed activities resulting in
false-negative errors where belongsToSameActivity relation-
ships stated by the test subjects were missing in the sensor
results.
In addition to the results shown in table 3 the document in-
terrelations being identified as false-positives showed a lower
Table 3: Evaluation Results
Test Subject Window
In-
stances
False
Nega-
tives
False
Posi-
tives
Error
Rate
1 A 62 1 6 11.29%
2 A 6 0 0 0.00%
3 A 12 0 1 8.33%
4 A 70 1 0 1.43%
5 B 12 0 1 8.33%
6 B 52 0 3 5.77%
7 B 28 0 0 0.00%
8 B 33 0 1 3.03%
9 B 21 0 2 9.52%
10 B 71 2 0 2.82%
11 C 51 0 0 0.00%
12 C 67 1 1 2.99%
13 D 16 0 0 0.00%
14 D 24 1 1 8.33%
15 D 60 0 1 1.67%
Total 585 6 17 3.93%
correctly identified relationships. Especially when consider-
ing that the reliabilities attached to document interrelations
are not discarded but made available to further processing
within the resulting semantic network a reconnection of the
previously fragmented document based information can be
achieved using the presented approach. So while the number
of participants in this evaluation was limited the error rate
lies in an acceptable range.
5. CONCLUSION AND OUTLOOK
It has been shown that the fragmentation of document based
information in enterprise environments slows down work-
flow related information retrieval and poses a significant
challenge to knowledge management initiatives. The pa-
per demonstrated a new method to relate such unstruc-
tured documents containing similar information to each
other without imposing additional tagging or training work-
load on the individual user. The method was prototypically
implemented and evaluated in a small scale testing setup.
The amount of combined errors during the performed eval-
uation were in a noticeable yet – for the purpose of the pre-
sented approach – acceptable range. While the arguably
more sophisticated existing task switching detection ap-
proaches as described in section 2 show a slightly higher
rate of error when detecting task and activity switches this
has to be explained by the steeper requirements applied by
them. For the purpose of interrelating documents which
does not require the detection of long running tasks contexts
taking possibly multiple days to complete the presented ap-
proach is sufficient though. Furthermore the presented ap-
proach has the advantage of not causing user workload by
imposing manual tagging or training requirements. Lastly
the generated semantic relationships as well as the attached
provenance and reliability statements can be further utilized
by third party semantic Personal Information Management
systems or Enterprise Content Management solutions.
In future research the mentioned heuristics to form activ-
ity clusters will have to be further tested and improved to
reduce the number of false-positives. To this end domain
specific heuristics might prove to be valuable. Also legisla-
tive aspects, especially privacy concerns of employees, have
to be considered. Semantically white- and/or blacklisting
named business entities as well as excluding specific window
types or file system paths from the initial monitoring might
be feasible to specifically include business process relevant
documents only or to exclude documents and communica-
tion of sensible parties from the context readings.
6. REFERENCES
[1] S. Bailey. Has EDRMS been a success? The case for
the prosecution. In Opening Statement to the RMS
Conference 2008, Edinburgh, 2008.
[2] V. Bellotti, N. Ducheneaut, M. Howard, and I. Smith.
Taking email to task: the design and evaluation of a
task management centered email tool. In Proceedings
of the SIGCHI conference on Human factors in
computing systems, page 352. ACM, 2003.
[3] R. Benbunan-Fich, R. Adler, and T. Mavlanova.
Towards new metrics for multitasking behavior. In
Proceedings of the 27th international conference
extended abstracts on Human factors in computing
systems, pages 4039–4044. ACM, 2009.
[4] O. Bergman, R. Beyth-Marom, and R. Nachmias. The
project fragmentation problem in personal information
management. In Proceedings of the SIGCHI conference
on Human Factors in computing systems, page 274.
ACM, 2006.
[5] R. Blumberg and S. Atre. The problem with
unstructured data. DM REVIEW, 13:42–49, 2003.
[6] H. Bru¨gmann. Assisting the Discovery and Reuse of
Document- based Knowledge using Semantic
Metadata. In M. Schumann, L. M. Kolbe, M. H.
Breitner, and A. Frerichs, editors, Multikonferenz
Wirtschaftsinformatik 2010, pages 27–38, Go¨ttingen,
2010. Universita¨tsverlag Go¨ttingen.
[7] D. Buttler. A short survey of document structure
similarity algorithms. Proceedings of the 5th
international conference on internet computing
(IC2004), pages 191–208, 2004.
[8] P. Castells, M. Fernandez, and D. Vallet. An
Adaptation of the Vector-Space Model for
Ontology-Based Information Retrieval. IEEE
Transactions on Knowledge and Data Engineering,
19(2), 2007.
[9] E. Cutrell, D. C. Robbins, S. T. Dumais, and
R. Sarin. Fast, Flexible Filtering with Phlat? Personal
Search and Organization Made Easy. Microsoft
Research, 2006.
[10] M. Czerwinski, G. Smith, T. Regan, B. Meyers,
G. Robertson, and G. Starkweather. Toward
characterizing the productivity benefits of very large
displays. In Human-computer interaction:
INTERACT’03; IFIP TC13 International Conference
on Human-Computer Interaction, 1st-5th September
2003, Zurich, Switzerland, page 9. Ios Pr Inc, 2003.
[11] A. Dragunov, T. Dietterich, K. Johnsrude,
M. McLaughlin, L. Li, and J. Herlocker. TaskTracer: a
desktop environment to support multi-tasking
knowledge workers. In Proceedings of the 10th
international conference on Intelligent user interfaces,
pages 75–82. ACM, 2005.
[12] S. Dumais, E. Cutrell, J. Cadiz, G. Jancke, R. Sarin,
and D. C. Robbins. Stuff I’ve seen: A system for
personal information retrieval and re-use. Annual
ACM Conference on Research and Development in
Information Retrieval, 2003.
[13] Ferris Research. Industry Statistics 2009,
http://www.ferris.com/research-library/industry-
statistics/,
04/07/2010.
[14] W. Geyer, M. J. Muller, M. T. Moore, E. Wilcox,
L.-T. Cheng, B. Brownholtz, C. Hill, and D. R.
Millen. Activity explorer: activity-centric
collaboration from research to product. IBM Systems
Journal, 45(4):713, 2006.
[15] M. R. Gilbert, K. M. Shegda, D. Logan, K. Chin,
T. Bell, L. Latham, R. E. Knox, and J. Lundy. Key
Issues for Enterprise Content Management, 2006.
Technical Report G00139515, Gartner Inc., 2006.
[16] V. M. Gonzalez and G. Mark. Constant, constant,
multi-tasking craziness: managing multiple working
spheres. In Proceedings of the ACM Conference on
ACM Press, 2004.
[17] M. Granitzer, G. Granitzer, K. Tochtermann,
S. Lindstaedt, A. Rath, and W. Groiß. Automating
Knowledge Transfer and Creation in Knowledge
Intensive Business Processes. In Proceedings of the
First Workshop on Business Process Management and
Social Software BPMS208 in conjunction with 6th
International Conference on Business Process
Management, pages 1–10. Springer Berlin /
Heidelberg, 2008.
[18] D. R. Hutchings, G. Smith, B. Meyers, M. Czerwinski,
and G. Robertson. Display space usage and window
management operation comparisons between single
monitor and multiple monitor users. ACM Press, New
York, New York, USA, 2004.
[19] W. Jones and H. Bruce. A Report on the
NSF-Sponsored Workshop on Personal Information
Management, 2005.
[20] W. Jones, S. Dumais, and H. Bruce. Once found, what
then? A study of keeping behaviors in the personal
use of Web information. Society for Information, 2002.
[21] V. Kaptelinin. UMEA: User-monitoring environment
for activities. In Proceedings of the SIGCHI conference
on Human factors in computing systems, volume 4,
page 1, Ft. Lauderdale, 2002. ACM Press.
[22] K. Kuutti. Activity theory as a potential framework for
human-computer interaction research, chapter 2, pages
17–44. Context and consciousness: Activity theory
and human-computer interaction. MIT Press, 1996.
[23] R. Lokaiczyk and M. Goertz. Extending Low Level
Context Events by Data Aggregation. Proceedings of
IKNOW Sˇ08 and IMEDIA 08, pages 118–125, 2008.
[24] D. Lowd and N. Kushmerick. Using salience to
segment desktop activity into projects. In IUI ’09:
Proceedings of the 14th International Conference on
Intelligent User Interfaces, page 463, New York, New
York, USA, 2009. ACM Press.
[25] P. Lyman and H. R. Varian. How Much Information
2003. Technical report, University of California -
Berkeley’s School of Information Management and
Systems, 2003.
[26] W. Money and A. Turner. Assessing knowledge
management system user acceptance with the
Technology Acceptance Model. International Journal
of Knowledge Management, 1(1):8–26, 2005.
[27] D. Oard and J. Kim. Modeling information content
using observable behavior. In Proceedings of the 64th
Annual Conference of the American Society for
Information Science and Technology, pages 481–488,
2001.
[28] N. Oliver, G. Smith, C. Thakkar, and A. C.
Surendran. SWISH: semantic analysis of window titles
and switching history. In Proceedings of the 11th
international conference on Intelligent user interfaces,
volume 06, page 201. ACM, 2006.
[29] M. Perez-Quinones, M. Tungare, P. Pyla, and
M. Kurdziolek. A Special Topics Course on Personal
Information Management, 2006.
[30] A. S. Rath. User Interaction Context - Studying and
Enhancing Automatic User Task Detection on the
ComputerDesktop via an Ontology-based User
Interaction Context Model. Dissertation, Graz
University of Technology, 2009.
[31] A. S. Rath, D. Devaurs, and S. N. Lindstaedt. UICO:
An Ontology-Based User Interaction Context Model
for Automatic Task Detection on the Computer
Desktop. In Proceedings of the 1st Workshop on
Context, Information and Ontologies - CIAO ’09,
pages 1–10, New York, New York, USA, 2009. ACM
Press.
[32] D. D. Salvucci, N. A. Taatgen, and J. P. Borst.
Toward a unified theory of the multitasking continuum.
ACM Press, New York, New York, USA, 2009.
[33] J. Shen, J. Irvine, X. Bao, M. Goodman, S. Kolibaba,
A. Tran, F. Carl, B. Kirschner, S. Stumpf, and
T. Dietterich. Detecting and correcting user activity
switches: Algorithms and interfaces. In Proceedings of
the 13th international conference on Intelligent user
interfaces, pages 117–126. ACM, 2009.
[34] M. Tungare, P. S. Pyla, M. Sampat, and P.-Q. nones.
M.: Syncables: A framework to support seamless data
migration across multiple platforms. In IEEE
International Conference on Portable Information
Devices, 2007.
[35] L. Uden, P. Valderas, and O. Pastor. An
activity-theory-based model to analyse Web
application requirements. Information Research,
13(2):1, 2008.
[36] C. J. Van Rijsbergen, S. E. Robertson, and M. F.
Porter. New models in probabilistic information
retrieval. In British Library Research and Development
Report no 5587, 1980.
[37] W. E. Winkler. The state of record linkage and current
research problems. Statistical Research Division US
Bureau of the Census Wachington DC, 1999.
[38] E. Zadok, J. Osborn, A. Shater, C. Wright, and K.-K.
Muniswamy-Reddy. Reducing Storage Management
Costs via Informed User-Based Policies, 2004.
Sign up today - FREE
Mendeley saves you time finding and organizing research. Learn more
- All your research in one place
- Add and import papers easily
- Access it anywhere, anytime


