Reclaiming informed consent to train mental health AI with patient data

Abstract

Companies increasingly train AI models on therapy transcripts, often relying on terms of service that obscure data use to obtain assent. Because AI systems generate inferences from data and enable unforeseen secondary uses, existing consent models do not adequately protect autonomy and confidentiality. We argue for (1) separate and explicit opt-in consent to train on patient data and (2) patient-led governance to prevent the recurrence of exploitation in medical innovation.