Exploring and Evaluating Different Game Mechanics for Anti-Phishing Learning Games

Abstract

Anti-phishing learning games are a promising approach for teaching end-users about phishing, as they offer scalable, engaging learning environments. Existing games have been criticized for limited game mechanics that do not allow detailed assessments of players’ acquired knowledge, instead focusing mostly on factual and conceptual knowledge to remember or understand. With the aim of evaluating the effects of new game mechanics on the classification performance of URLs as phishing or benign, and on the understanding of in-game decisions, this paper presents the design and evaluation of two new learning games targeted at end-users who do not necessarily have previous knowledge of IT security: The first game implements extended classification mechanics to better assess players’ decision processes, while the second game implements different mechanics, asking players to combine URL parts when creating their own phishing URLs. In a case study with 133 participants, we compared the games with each other and with a third baseline game using binary decisions similar to related work. The study shows, that while all games lead to performance increases, new games do not offer sig-nificant improvements over the baseline. Longitudinal tests three months later show that knowledge can be retained as participants still performed significantly better than before playing either of the games.