Mapping the Iceberg: The Impact of Data Sources on the Study of District Courts

Abstract

Three decades ago, Siegelman and Donohue aptly characterized research about courts and litigation that relied only on published opinions as “studying the iceberg from its tip.” They implored researchers to view published district court opinions “with greater sensitivity to the ways in which such cases are unrepresentative of all cases”. The dynamic, multistage nature of trial court litigation makes a focus solely on published opinions particularly ill-suited to the study of federal district courts. Expanded electronic access to court documents now allows more precise analysis of the ways in which published cases are unrepresentative and what differences that makes for conclusions about the work of district courts. Heeding Siegelman and Donohue's admonition, this study seeks to map the iceberg, exploring the extent to which the visible part misrepresents what lies below the surface. Using a supplemented version of the Kim, Schlanger, and Martin EEOC Litigation Project data, this article examines the varying extent to which cases and judicial activity are visible in the several data sources commonly used by district court researchers. More specifically, we analyze how the work of federal district courts looks different depending on whether research relies on published opinions, on opinions available on Westlaw or Lexis (both “published” and “unpublished”), or on more comprehensive data available on PACER (Public Access to Court Electronic Documents). Our results reveal vast variation in visibility of cases and motions, depending on the data source used. We also demonstrate that these differences in case and motion visibility can affect the results of empirical analyses relating to, for example, the success rates of litigants and whether the party of the appointing president affects judicial behavior. Our findings mean that utilizing docket sheets, now available electronically, to gather data will often be required to draw accurate conclusions about the nature of district court litigation and the behavior of district court judges.