Sociotechnical design in legal algorithmic decision-making

Abstract

Over the past decade, civil litigants in the U.S. have come to increasingly rely on machine learning (ML) systems to classify documents for discovery review and fact-finding, an approach now broadly referred to as Technology-Assisted Review (TAR). The transformation of legal discovery from a painstaking manual process to a sophisticated algorithm-driven methodology took place over a relatively short period of time, many years before controversies arose surrounding the use of automated risk assessment tools on the criminal side of the U.S. justice system. Introduced in 2008 to a handful of litigators in an experimental research setting hosted by the National Institute of Standards and Technology (NIST), TAR was first deployed live on an active litigation in 2012, and by 2015 a vocal and influential vanguard of judges was actively advocating for its use on cases involving large, complex document discovery. My research examines the cross-disciplinary experimentation and collaboration that took place across legal practitioners and computer scientists leading to ML becoming a judicially accepted solution in U.S. civil litigation practice. The aim of this research is to develop a comprehensive case study for how an expert professional field wrestled with the challenges of integrating ML into sensitive decision-making workflows. While deeply attentive to the unique and complex encounter between U.S. civil litigation practice and computer science, my work also aims to inform the practice of algorithmic system design, development, and governance across other high-stakes professional domains.