Social Conjuring: Multi-User Runtime Collaboration with GenAI in Building Virtual Reality Worlds

Abstract

We present Social Conjurer, a system and framework that enables real-time, AI-augmented creation of virtual 3D environments for multiple users. Unlike prior GenAI systems that focus on single-user workflows or static scenes, Social Conjurer allows co-located or remote users to collaboratively build worlds using natural language, sketches, and tool-based interactions. We integrate LLMs, VLMs, and a custom networked Unity architecture to support spatial reasoning, dynamic asset placement, and shared editing. A study with 12 participants suggests the system enables co-creative worldbuilding.