Toward Independent Online Shopping of the Visually Impaired Through Voice-based Computer-Using Agent

Abstract

Visually impaired individuals face barriers in online shopping because product details are often conveyed through images, and alternative text is frequently insufficient. The recent advent of Computer-Using Agents (CUA) based on Large Multimodal Models, which can directly manipulate graphical user interfaces, offers new opportunities for such accessibility. However, there is a lack of research that considers how voice-based systems should be designed to support visually impaired users in complex online shopping contexts. Thus, our study qualitatively explores the experiences and needs of visually impaired users as they shop online through voice interaction with a CUA We conducted a Semi-Automatic Wizard-of-Oz study with 12 visually impaired participants, in which they performed shopping tasks with a voice-based CUA system, followed by debriefing interviews. This paper proposes design implications for creating a more inclusive and disability-centered online shopping environment with voice-based CUA for visually impaired users.