Auditing Work: Exploring the New York City algorithmic bias audit regime

Abstract

In July 2023, New York City (NYC) implemented the first attempt to create an algorithm auditing regime for commercial machine-learning systems. Local Law 144 (LL 144), requires NYC-based employers using automated employment decision-making tools (AEDTs) in hiring to be subject to annual bias audits by an independent auditor. In this paper, we analyse what lessons can be learned from LL 144 for other national attempts to create algorithm auditing regimes. Using qualitative interviews with 17 experts and practitioners working within the regime, we find LL 144 has failed to create an effective auditing regime: the law fails to clearly define key aspects like AEDTs and what constitutes an independent auditor, leaving auditors, vendors who create AEDTs, and companies using AEDTs to define the law's practical implementation in ways that failed to protect job applicants. Several factors contribute to this: first, the law was premised on a faulty transparency-driven theory of change that fails to stop biased AEDTs from being used by employers. Second, industry lobbying led to the definition of what constitutes an AEDT being narrowed to the point where most companies considered their tools exempt. Third, we find auditors face enormous practical and cultural challenges gaining access to data from employers and vendors building these tools. Fourth, we find wide disagreement over what constitutes a legitimate auditor and identify four different kinds of 'auditor roles' that serve different functions and offer different kinds of services. We conclude with four recommendations for policymakers seeking to create similar bias auditing regimes that use clearer definitions and metrics and more accountability. By exploring LL 144 through the lens of auditors, our paper advances the evidence base around audit as an accountability mechanism, and can provide guidance for policymakers seeking to create similar regimes.