Virtual Triplets: A Mixed Modal Synchronous and Asynchronous Collaboration with Human-Agent Interaction in Virtual Reality

Abstract

We introduce the Virtual Triplets (VTs), a Virtual Reality collaborative system designed for both synchronous and asynchronous interaction through collaboration between humans and human-agent. When a single instructor supervises two students in a showcased virtual cooking class scenario, the system enables the instructor to switch control between avatars in two separate environments, and a virtual agent takes over in the instructor's absence, ensuring continuous support. VTs facilitate the recording and playback of the instructor's demonstrations for students, coupled with a feature that allows the instructor to employ a bird's-eye view for effective classroom management. Analysis of observational data and interviews from our pilot study indicates good system usability, ease of avatar management for instructors, and a strong sense of continuous support among students. We discuss potential improvements and broader applications of VTs, aiming to enhance user experience in multitasking scenarios involving multiuser human-human and human-agent collaboration.