VirtualEnv: A Platform for Embodied AI Research

TL;DR

VirtualEnv addresses the need for realistic, interactive simulators to rigorously evaluate embodied AI and LLM-driven agents. It couples Unreal Engine 5–based photorealism with a modular scene-graph and a language-driven API, enabling procedural task generation and real-time environment editing via vLLMs and VLMs. The paper introduces an Escape Room benchmark with escalating cognitive demand and reports that reasoning-enabled LLMs outperform baselines, especially on complex tasks, while multi-agent collaboration enhances performance. Open-sourcing VirtualEnv aims to standardize multimodal embodied AI benchmarks and accelerate research at the intersection of AI, simulation, and immersive gaming technologies.

Abstract

As large language models (LLMs) continue to improve in reasoning and decision-making, there is a growing need for realistic and interactive environments where their abilities can be rigorously evaluated. We present VirtualEnv, a next-generation simulation platform built on Unreal Engine 5 that enables fine-grained benchmarking of LLMs in embodied and interactive scenarios. VirtualEnv supports rich agent-environment interactions, including object manipulation, navigation, and adaptive multi-agent collaboration, as well as game-inspired mechanics like escape rooms and procedurally generated environments. We provide a user-friendly API built on top of Unreal Engine, allowing researchers to deploy and control LLM-driven agents using natural language instructions. We integrate large-scale LLMs and vision-language models (VLMs), such as GPT-based models, to generate novel environments and structured tasks from multimodal inputs. Our experiments benchmark the performance of several popular LLMs across tasks of increasing complexity, analyzing differences in adaptability, planning, and multi-agent coordination. We also describe our methodology for procedural task generation, task validation, and real-time environment control. VirtualEnv is released as an open-source platform, we aim to advance research at the intersection of AI and gaming, enable standardized evaluation of LLMs in embodied AI settings, and pave the way for future developments in immersive simulations and interactive entertainment.

Figures (5)

• Figure 2: Core capabilities of VirtualEnv. The platform supports highly realistic and interactable indoor and outdoor environments, a large curated library of over 20,000 diverse assets, and controllable humanoid agents with fine-grained motion support. These features enable the creation of complex, multimodal simulation scenarios suitable for embodied AI training, evaluation, and benchmarking.
• Figure 3: Language-based task and scenario generation in VirtualEnv. A user provides a natural language prompt describing a high-level scenario, which is parsed by a vLLM into sub-tasks and structured goals (e.g., solving riddles, unlocking containers). Based on the task requirements, VirtualEnv automatically evaluates which objects are needed, updates the scene graph accordingly, and renders the environment through the VirtualEnv API. This pipeline enables flexible and scalable creation of interactive, goal-driven scenarios without manual scripting.
• Figure 4: Interactive environment editing and interpretation validation in VirtualEnv. Users provide natural language instructions to modify the environment (e.g., adding, replacing, or removing objects). A vLLM interprets these instructions and updates the scene graph, which is then rendered using the VirtualEnv API. To ensure semantic alignment between the symbolic scene graph and the visual output, the system performs interpretation checks on both the JSON graph and the rendered image. This process enables reliable, language-driven environment manipulation and validation.
• Figure 5: Comparison of Visual Realism Rankings Across Platforms. A qualitative benchmarking study was conducted using a survey with 31 respondents. Participants ranked each platform based on visual realism, assigning a score from 5 (most realistic) to 1 (least realistic) in a label-blind test. We observe that the participants consider VirtualEnv to be significantly more visually realistic than all other environments.
• Figure 6: Distribution of Failure Modes in Embodied AI Tasks. Analysis of failure modes reveals six primary categories.