RePlay: Touchscreen Interaction Substitution Method for Accessible Gaming

Abstract

We present RePlay, an interaction substitution method designed to support people with upper extremity motor impairments while interacting with mobile apps. RePlay overcomes key limitations of existing approaches, allowing users to quickly access all app interface elements, even those not visible to the OS (a common situation especially with games). This is achieved through personalized mapping of interface elements to alternative inputs, such as external switches or non-verbal vocal sounds, hence adapting to users with diverse abilities. RePlay was implemented as an Android Accessibility Service, running without any changes at OS level. An evaluation conducted with ten participants with upper extremity motor impairments shows that RePlay can provide access to mobile games that would otherwise be inaccessible. Still, reaction time using RePlay is slower than direct touchscreen interaction by users without motor impairments, and therefore speed compensation might be needed. Both external switches and non-verbal voice input achieve comparable reaction times, while their combination is slower and more cognitively demanding. Prolonged voice input may also cause fatigue and should be used parsimoniously.