Evaluating the stability and credibility of ontology matching methods

Abstract

Ontology matching is one of the key research topics in Semantic Web. In the last few years, many matching methods have been proposed to generate matches between different ontologies either automatically or semi-automatically. To select appropriate ones, users need some measures to judge whether a method can achieve the similar compliance even on one dataset without reference matches and whether such a method is reliable w.r.t. its output result along with the confidence. However, widely-used traditional measures like precision and recall fail to provide sufficient hints. In this paper, we design two novel evaluation measures to evaluate stability of matching methods and one measure to evaluate credibility of matching confidence values, which help answer the above two questions. Additionally, we carry out comparisons among several carefully selected methods systematically using our new measures. Besides, we report some interesting findings such as identifying potential defects of our subjects. © 2011 Springer-Verlag Berlin Heidelberg.