Measuring effectiveness of ontology debugging systems

Abstract

Ontology debugging aims to provide users with justifications for an entailment in OWL ontologies. So far, many ontology debugging algorithms have been proposed and several ontology debugging systems are available. There has been some work on evaluating these systems with the efficiency as the main evaluation measure. However, existing systems may fail to find all justifications for an entailment within a time limit and may return incorrect justifications. Therefore, measuring their effectiveness by considering the correctness of justifications and the completeness of a found set of justifications is helpful. In this paper, we first give a survey of existing ontology debugging approaches and systems. We then evaluate both the effectiveness and the efficiency of existing ontology debugging systems based on a large collection of diverse ontologies. To assess the effectiveness of an ontology debugging system, we first propose a method to construct the reference justification sets and define the degrees of correctness and completeness of the system. Then we construct a dataset containing 80 ontologies with significantly different sizes and expressivities. Based on the proposed evaluation measures and the constructed dataset, we do comprehensive experiments. The results show the advantages and disadvantages of existing ontology debugging systems in terms of correctness, completeness and efficiency. Based on the results, we provide several suggestions for users to choose an appropriate ontology debugging system and for developers to design an ontology debugging algorithm and build an ontology debugging system.