The Devil is in the Details: Interrogating Values Embedded in the Allegheny Family Screening Tool

Abstract

The design decisions of developers and researchers in creating algorithmic tools - like constructing variables, performing feature selection, and binning model outputs - are sometimes cast as objective technical processes. In reality, these decisions are far from objective, and they are sometimes even made arbitrarily. In this work, we examine how algorithmic design choices can function as policy decisions through an audit of a deployed algorithmic tool, the Allegheny Family Screening Tool (AFST), used to screen calls to a child welfare agency about alleged child neglect in Allegheny County, Pennsylvania. We analyze design decisions in the AFST's development process related to feature selection, data collection, and post-processing, highlighting three values implicitly embedded in the tool through these decisions. By aggregating risk scores at the household level, the AFST effectively treats families as "risky"by association. In choosing to use training data from the criminal legal system and behavioral health agencies, the AFST prioritizes "making decisions based on as much information as possible,"even when that information is potentially biased across race, disability, and other protected statuses. Finally, by including static features in the model that identify whether a person has ever been affected by the criminal legal system or relied on public benefits, the AFST chooses to mark families in perpetuity, compounding the impacts of systemic discrimination and foreclosing opportunities for recourse for families impacted by the tool. We explore the impacts of these decisions, individually and together, arguing that they function as policy choices that may have discriminatory effects and raise concerns about lack of democratic oversight.