Geno: A developer tool for authoring multimodal interaction on existing web applications

Abstract

Supporting voice commands in applications presents significant benefits to users. However, adding such support to existing GUI-based web apps is effort-consuming with a high learning barrier, as shown in our formative study, due to the lack of unified support for creating multi-modal interfaces. We develop Geno - -a developer tool for adding the voice input modality to existing web apps without requiring significate NLP expertise. Geno provides a unified workflow for developers to specify functionalities to support by voice (intents), create language models for detecting intents and the relevant information (parameters) from user utterances, and fulfill the intents by either programmatically invoking the corresponding functions or replaying GUI actions on the web app. Geno further supports references to GUI context in voice commands (e.g., "add this to the playlist"). In a study, developers with little NLP expertise were able to add the multi-modal support for two existing web apps using Geno.