Challenges and Opportunities on the Application of Heuristic Evaluations: A Systematic Literature Review

Abstract

Heuristic evaluation belongs to the usability inspection methods and is considered one of the most popular methods since it allows to discover over 75% of the total usability problems involving only 3 to 5 usability experts, in comparison with user tests. However, certain problems and challenges have been identified at the time of their execution. In this study we present the results of conducting a Systematic Literature Review (SLR) to identify case studies, challenges, problems, and opportunities on the execution of heuristic evaluations in the context of a research for the automation and formalization of the process. For this SLR, we have employed the protocol proposed by Kitchenham and Charters. The research was carried out on September 7 of 2020 and retrieved a total of 167 studies of which 37 were selected for this review. The results show that the main challenges are related to the low suitability of the chosen set of heuristics and the low expertise of usability evaluators. Additionally, we have identified that exist very few software solutions that support and automate the process. Finally, we have found that there were many protocols to follow when applying a heuristic evaluation, like the definition of new usability heuristics for a given case of study. According to the results obtained, we can conclude that it is necessary to develop and validate a tool based on a formal protocol that supports and automate the heuristic evaluation process, that gives solutions to the challenges and opportunities identified in this research.