COLIEE-2018: Evaluation of the Competition on Legal Information Extraction and Entailment

Abstract

We summarize the evaluation of the 5th Competition on Legal Information Extraction/Entailment 2018 (COLIEE-2018). The COLIEE-2018 tasks include two tasks in each of statute law and case law. The case law component includes an information retrieval (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes information retrieval (Task 3) and entailment/question answering (Task 4). Participation was open to any group based on any approach. 13 teams participated in the case law competition, and we received results from 7 teams where 6 submissions to Task 1 (12 runs), and 4 submissions to Task 2 (8 runs). Regarding the statute law, there were submissions of 17 runs from 8 teams (including 2 organizers’ runs) for Task 3 and 7 runs from 3 teams for Task 4. We describe each team’s approaches, our official evaluation, and analysis on our data and submission results. We also discuss possibilities for future competition tasks.