Process mining crimes – A threat to the validity of process discovery evaluations

Abstract

Given the multitude of new approaches and techniques for process mining, a thorough evaluation of new contributions has become an indispensable part of every publication. In this paper, we present a set of 20 scientifically supported “process mining crimes”, unintentional mistakes that threaten the validity of process discovery evaluations. To determine their prevalence even in high-quality publications, we perform a meta-evaluation of 21 process discovery papers published at the BPM conference. We find that none of these papers is completely crime-free, but the number of crimes and their impact on the evaluations’ validity differs considerably. Based on our list of crimes, we suggest a catalog of 13 process mining guidelines, which may contribute to avoiding process mining crimes in future evaluations. Our objective is to spark an open discussion about the necessity of valid evaluation results among both process mining researchers and practitioners.