Precision, Fitness, Generalization, and Simplicity as Quality Dimensions for Decision Discovery Algorithms

Abstract

Featured Application: The proposed precision, fitness, generalization, and simplicity metrics for decision mining can be directly applied in domains where transparency and accountability of decision-making are important. For example, in public administration, decision discovery algorithms can be evaluated and improved when applied to regulatory contexts. By assessing whether a discovered decision model is both precise and fit, organizations can ensure that automated or semi-automated decision-making processes remain aligned with legal requirements and public values. Beyond regulatory compliance, the metrics provide a practical tool for practitioners to identify over- and under-specifications in decision logic, enabling an iterative refinement of decision models. This facilitates not only more reliable operational decisions but also contributes to broader societal goals such as fairness and trust in digital decision-making systems. Operational decisions significantly influence organizational performance and individual well-being. Decision mining offers a method to discover and analyze decision logic from decision logs, enhancing decision-making processes. However, evaluating the quality of decision discovery algorithms remains a challenge. While precision, fitness, generalization, and simplicity are well-established quality dimensions in process mining, their adaptation to the decision mining domain is underexplored. This study adapts these four dimensions to the necessary characteristics of decision models, providing a framework for evaluating decision discovery algorithms. Using a design science research approach, we develop tailored metrics and functions and demonstrate their application through a practical example of environmental permit management modeled in Decision Model and Notation (DMN). Precision measures how the discovered decision model reproduces the observed fact types and values from the decision log, detecting over-specification in the decision model. Fitness evaluates how completely the decision model covers the behavior in the decision log, identifying missing or under-specified elements in the decision model. Generalization assesses the model’s robustness to unseen decision cases by quantifying how well the discovered decision model performs beyond the training data. Simplicity captures the complexity in the discovered decision model in relation to a human actor-specified threshold. These insights guide decision model improvements, contributing to higher transparency, accountability, and fairness in operational decision-making processes. This research bridges a gap in the body of knowledge by providing a concrete methodology for evaluating decision discovery algorithms. The results support organizations in aligning decision models with regulatory requirements and public values, while also laying a foundation for future research.