Holodeck: Case Studies in Exploring AI Use in Automatic VR World Generation

Recent advancements in AI, particularly through transformer and diffusion architectures, have unlocked possibilities once considered science fiction, such as generating 3D objects from simple prompts and creating interactive avatars capable of dynamic responses and role-playing. This paper presents three case studies examining different workflows for leveraging the context of objects or avatars to assign specific properties, with the vision of enabling the development of immersive applications within LLM-powered virtual environments. The case studies explore how object properties, such as scale and relational position, can be defined and how agent-driven avatars can reflect their capabilities and select animations, while outlining challenges for future developers, including refining prompt design for conciseness and inference speed, diversifying output types, and establishing objective evaluation methods. Using the open-source Holodeck project as a basis, these concepts are demonstrated through three case studies: one on object scaling, animations, and reflective behaviors of a teaching assistant avatar, and finally a workflow and LLM benchmark for exploring objects’ relational properties to each other. This work aims to highlight the potential for integrating contextual properties into generated virtual worlds, allowing non-expert practitioners to help build VR applications in the future.