Interactive, multi-device visualization supported by a multimodal interaction framework: Proof of concept

Abstract

Nowadays, users can interact with a system using a wide variety of modalities, such as touch and speech. Nevertheless, multimodal interaction has yet to be explored for interactive visualization scenarios. Furthermore, users have access to a wide variety of devices (e.g., smartphones, tablets) that could be harnessed to provide a more versatile visualization experience, whether by providing complementary views or by enabling multiple users to jointly explore the visualization using their devices. In our effort to gather multimodal interaction and multi-device support for visualization, this paper describes our first approach to an interactive multi-device system, based on the multimodal interaction architecture proposed by the W3C, enabling interactive visualization using different devices and representations. It allows users to run the application in different types of devices, e.g., tablets or smartphones, and the visualizations can be adapted to multiple screen sizes, by selecting different representations, with different levels of detail, depending on the device characteristics. Groups of users can rely on their personal devices to synchronously visualize and interact with the same data, maintaining the ability to use a custom representation according to their personal needs. A preliminary evaluation was performed, mostly to collect users’ first impressions and guide future developments. Although the results show a moderate user satisfaction, somehow expected at this early stage of development, user feedback allowed the identification of important routes for future improvement, particularly regarding a more versatile navigation along the data and the definition of composite visualizations (e.g., by gathering multiple representations on the same screen).