Stopping Criteria for Technology Assisted Reviews based on Counting Processes

Abstract

Technology Assisted Review (TAR) aims to minimise the manual judgements required to identify relevant documents. Reductions in workload are dependent on a reviewer being able to make an informed decision about when to stop examining documents. Counting processes offer a theoretically sound approach to creating stopping criteria for TAR approaches that are based on analysis of the rate at which relevant documents are observed. This paper introduces two modifications to existing approaches: application of a Cox Process (a counting process which has not previously been used for this problem) and use of a rate function based on a power law. Experiments on the CLEF 2017 e-Health TAR collection demonstrates that these approaches produces results that are superior to those reported previously.