Human-First, Please: Assessing Citizen Views and Industrial Ambition for Emotional AI in Recommender Systems

Abstract

This paper qualitatively explores the views of diverse members of the British public on applications of biometric emotional AI technologies patented by two globally dominant consumer-facing recommender systems, Amazon and Spotify. Examining Amazon and Spotify patents for biometric profiling of users’ emotions, disposition, and behaviour to offer them tailored services, ads, and products from their wider platforms, this paper points to industrial ambition regarding emotional AI. Little is known about ordinary people’s views on deployment of such technology, and given the complex, abstract, and future-facing nature of such technologies, ascertaining informed lay views is hard. We address this through our innovative, qualitative study of diverse British-based adults (n=46) that presents to them near-horizon use cases in an interactive fictional narrative that deploys design fiction principles and ContraVision techniques. We find the themes of “usefulness,” “resignation,” “uneasy terms of engagement,” and “human-first,” adding rich and nuanced insights to prior survey work on users’ views towards biometric-based emotional AI technologies. In contributing to a richer understanding of whether emotional AI technologies should be deployed in consumer-facing recommender systems, and if so, on what terms, we find that well-established policy-friendly criticisms apply to global emotional AI recommender systems. We conclude, however, that problems of alienation and need for a human-first approach to emerging AI technology are the most significant criticisms.