What’s Wrong with OWL Benchmarks?

Abstract

Abstract. This paper is motivated by experiences in course of developing an ontology-based application within a real-world setting. When comparing these experiences with results of well-known benchmarks we found out that current benchmarks are not well suited to provide helpful hints for application developers who seek for a reasoning system matching typical real-world needs. This work aims at identifying requirements to make future benchmarks more useful for application developers. 1 On Benchmarking OWL Reasoners Reasoning with OWL Lite as well as OWL DL is known to be of high worstcase complexity. By using the “right ” combination of costly expressions, one can intentionally or incidentally create even a very small ontology whose complexity will make practical reasoning impossible. On the other hand, recent implementations have shown significant progress with respect to the scalability of instance retrieval over ABoxes containing several hundred thousands of individuals. These observations cannot be seen as separate issues. Quite the contrary,