Improving heuristic evaluation through collaborative working

Abstract

The standard method of heuristic evaluation finds many potential usability problems but as most are found by only one evaluator this indicates an overall low reliability of the method. This study investigated using a collaborative approach to improve the reliability of problems reported from an evaluation. The collaborative heuristic evaluation (CHE) method enables evaluators to experience the interface together but make independent judgements of whether a problem exists, and if so, the appropriate severity and heuristics. Four websites were evaluated in a between-groups repeated-measures design and CHE was found to have an inter-evaluator agreement of problems higher than 83%. Agreement of severity rating ranged between 32 and 45%, no agreement of heuristics use was found. Limitations of the present study are noted and implications for future research are discussed.