Evaluating Professional Search: A German Construction Law Use Case

Abstract

We present a real world case study for the evaluation of professional search focusing on German construction law. Reliable identification of relevant previous cases is an important part of many legal disputes, and currently relies on domain expertise acquired over a lengthy professional career. We describe our experiences from the development of a Cranfield type test collection for a German construction law dataset to enable research into the development of search technologies for new tools which are less dependent on expert knowledge. We describe examination of the search needs of lawyers, the development of a set of search queries created by lawyers, and our experiences in collecting expert relevance data for the completion of a test collection for legal search. Important findings of this latter process are the need for individuals with expert legal training to assess relevance, and the identification of context dependence in determining relevance. While the cost of the development of this test collection was found to be very high, we demonstrate its value in terms of identifying the effectiveness of legal search methods and in identifying research directions for legal case search.