When Generative Artificial Intelligence Meets Extended Reality: Enabling Scalable and Natural Interactions

Abstract

Extended reality (XR), including virtual, augmented, and mixed reality (VR/AR/MR), provides immersive and interactive experiences across diverse applications, from VR-based education to AR-based assistance and MR-based training. However, widespread XR adoption remains limited due to two key challenges: 1) the high cost and complexity of authoring 3-D content, especially for large-scale environments or complex interactions; and 2) the steep learning curve associated with nonintuitive interaction methods, like handheld controllers or scripted gestures. Generative artificial intelligence (GenAI) presents a promising solution by enabling intuitive, language-driven interaction and automating content generation. Leveraging vision–language models and diffusion-based generation, GenAI can interpret ambiguous instructions, understand physical scenes, and generate or manipulate 3-D content, significantly lowering barriers to XR adoption. This article explores the integration of XR and GenAI through three concrete use cases, showing how they address key obstacles in scalability and natural interaction, and identifying technical challenges that must be resolved to enable broader adoption.