87
Evaluation Criteria for Assessing the Usability of ERP
Systems
Akash Singh
Department of Computer Science
and Information Systems
Nelson Mandela Metropolitan
University
P.O. Box 77000
Port Elizabeth, South Africa, 6031
+27 (0) 12 999 9100
Akash.Singh@nmmu.ac.za
Janet Wesson
Department of Computer Science
and Information Systems
Nelson Mandela Metropolitan
University
P.O. Box 77000
Port Elizabeth, South Africa, 6031
+ 27 (0) 41 — 504 2323
Janet.Wesson@nmmu.ac.za

Abstract
It is widely acknowledged that enterprise resource planning (ERP)
systems suffer from complex user interfaces. The complexity of
these user interfaces negatively affects the usability of these
systems. Current research has shown that a need exists to improve
the overall usability of ERP systems. Specific methods and
criteria for assessing the usability of ERP systems have not been
developed or widely published. This paper proposes a set of
heuristics which can be used to assess the usability of ERP
systems. The proposed heuristics were verified by means of a case
study involving a heuristic evaluation of an ERP system.
Categories and Subject Descriptors
C.4 [Computer Systems Organization]: Performance of Systems
– Design Studies.
H.1.2 [Models and Principles]: User/Machine Systems – Human
Factors.
H.5.2 [Information Interfaces and Presentation]: User
Interfaces – Evaluations, Graphical User Interfaces.
J.1 [Computer Applications]: Administrative Data Processing –
Business, Manufacturing.
General Terms
Measurement, Human Factors, Verification.
Keywords
Enterprise Resource Planning Systems, Usability, Usability
Criteria, Usability Testing, Heuristics, User Interfaces.
1. Introduction
An enterprise resource planning (ERP) system can be defined as a
single integrated and packaged business information system. The
aim of an ERP system is to seamlessly integrate and manage the
different business processes and information flows within an
enterprise [1; 16; 24]. ERP systems have significant benefits to
offer an enterprise in terms of improving operational efficiency
[3]. Despite having significant benefits, ERP systems have been
criticized as being too complex.
This complexity is mostly due to these systems having to integrate
and process large amounts of data. This has resulted in ERP
systems possessing user interfaces (UIs) which suffer from poor
usability [20]. Usability problems (issues) can hamper the extent
to which a system can be used by its users to achieve a set of
goals within a specified context of use [9].
A need exists to provide UIs for ERP systems which are easy to
use, easy to learn and which support the users’ tasks. This need
has not been widely researched. Limited research is available
which highlights the usability issues of ERP systems. Identifying
the usability issues with existing ERP systems requires the use of
suitable usability evaluation methods and usability criteria which
are specific to ERP systems. The problem occurs whereby
existing methods and criteria used to evaluate the usability of ERP
system are inconsistent and often disparate.
The aim of this paper is to propose a set of usability heuristics that
are specific to ERP systems. These heuristics are intended to
facilitate the identification of usability issues in ERP systems. The
remainder of this paper comprises the following sections: Sections
2 and 3 contain a discussion of related work in terms of measuring
general system usability and the usability of ERP systems. A set
of ERP heuristics, which could be used to identify the usability
issues of an ERP system, is proposed in Section. 4. Section 5
presents a case study which discusses how the proposed set of
ERP heuristics was verified using a heuristic evaluation. This is
followed by conclusions (Section 6), acknowledgements (Section
7) and a list of references.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior specific permission and/or a fee.
SAICSIT’09,12-14 October 2009, Riverside, Vanderbijlpark, South
Africa.
Copyright 2009 ACM 978-1-60558-643-4…$5.00.

88
2. Measuring Usability
Usability can be measured empirically or analytically through the
use of inspection methods (techniques) [7; 8]. The empirical
approach to measuring usability attempts to identify actual
usability issues that are experienced by real users [7; 18].
Usability inspection is regarded as a cost-effective approach (to
measuring usability) which relies on the review of usability
experts [7]. Inspection techniques differ from empirical
techniques by identifying potential usability issues, as opposed to
actual usability issues. This is accomplished by having usability
experts analyze a UI with a set of guidelines or questions [5]. One
of the most popular inspection techniques is heuristic evaluation.
2.1 Heuristic Evaluations
A heuristic evaluation can be regarded as an informal technique of
measuring usability [14]. This inspection technique involves a
small number of usability experts (evaluators) evaluating the
usability of a UI and allocating a severity rating using a set of UI
guidelines or heuristics [7; 14]. Heuristic evaluations have been
identified as the most commonly used approach to measuring
usability due to the ease in which these evaluations can be
conducted and executed [21].
Some of the benefits of heuristic evaluations include [14; 21]:
1. Low-cost means to measuring usability;
2. Usability can be measured earlier in the development
process;
3. No advanced planning is required;
4. Evaluations can be conducted effectively without using
professional evaluators; and
5. The method of conducting a heuristic evaluation is easy to
learn.

There are three different ways in which a heuristic evaluation can
be conducted: using heuristic rules, a subjective judgment or a
task-based evaluation [11]. Heuristic rules allow for potential
usability issues to be identified by evaluating a system against a
set of rules of guidelines. The subjective judgment approach
requires evaluators to use their knowledge gained from previous
usability evaluations to assess the usability of a system. The task-
based method requires evaluators to examine how tasks are
carried out on the system and record any problems that they might
encounter.
Typically, between three to five evaluators are used to conduct the
heuristic evaluation [4; 14]. Research has shown that this number
of evaluators can identify 75% of all usability problems. When
there are more than ten evaluators, the results begin to diminish
[4; 14].
The process of conducting a heuristic evaluation is generally a
three-step process. Firstly, the evaluators are briefed in terms of
what needs to be done. Second is the evaluation period – the
period during which the evaluators inspect the system for possible
usability problems. This is followed by the debriefing session,
during which the experts come together and discuss their findings
in an attempt to suggest some possible solutions [11].
A set of usability heuristics is the basis of a heuristic evaluation.
2.2 Usability Heuristics
Usability heuristics can be defined as guidelines or general
principles that can guide a UI design decision or that can be used
to evaluate a decision that has already been made [4; 14; 21]. Two
of the most widely used set of usability heuristics are Nielson’s
ten heuristics [13] and Shneiderman’s eight golden rules for UI
design [19].
Nielson’s ten heuristics, which is the more popular of the two, are
[13]:
1. Visibility of system status;
2. Match between system and the real world;
3. User control and freedom;
4. Consistency and standards;
5. Error prevention;
6. Recognition rather than recall;
7. Flexibility and efficiency of use;
8. Aesthetic and minimalist design;
9. Help users recognize, diagnose and recover from errors;
and
10. Help and documentation.

Shneiderman’s eight golden rules for UI design are [19; 21]:
1. Strive for consistency;
2. Cater to universal usability;
3. Offer informative feedback;
4. Design dialogs to yield closure;
5. Prevent errors;
6. Permit easy reversal of actions;
7. Support internal locus of control; and
8. Reduce short-term memory load.

Typically, evaluators assess systems based on these heuristics and
identify any violation of these heuristics as a potential usability
issue. In order to determine the severity of the usability issue,
severity scales can be used. A typical severity scale is a likert
scale from 0 – 4, which is interpreted as follows [4]:
0. Not a usability problem;
1. Cosmetic problem: need not be fixed unless extra time
is available;
2. Minor usability problem: fixing this should be given a
low priority;
3. Major usability problem: important to fix, should be
given a high priority; and
4. Usability catastrophe: imperative to fix this problem
before product can be released.