Real2USD: Scene Representations in Universal Scene Description Language

TL;DR

The paper introduces Real2USD, a pipeline to convert real indoor scenes into Universal Scene Description USD scene graphs, enabling LLM-based reasoning and planning for robotics. By representing geometry, appearance, and semantics in a human- and LLM-readable XML-based hierarchy, Real2USD supports long-horizon tasks driven by language prompts, with recognition, localization, and reconciliation stages anchored in simulation for physical plausibility. Through real-world experiments with a Unitree Go2 and simulated studies in Isaac Sim, the approach demonstrates semantic task execution and improved 3D metric-semantic mapping compared with a baseline, highlighting the practical potential of USD as a universal, language-friendly scene representation. The work shows that USD can serve as a general lingua franca for robotics perception and planning, enabling more flexible, task-agnostic integration with LLMs and paving the way for broader adoption in real-world deployments. Overall, Real2USD advances a scalable, interpretable, and LLM-accessible pathway for semantic robotics in complex indoor environments.

Abstract

Large Language Models (LLMs) can help robots reason about abstract task specifications. This requires augmenting classical representations of the environment used by robots with natural language-based priors. There are a number of existing approaches to doing so, but they are tailored to specific tasks, e.g., visual-language models for navigation, language-guided neural radiance fields for mapping, etc. This paper argues that the Universal Scene Description (USD) language is an effective and general representation of geometric, photometric and semantic information in the environment for LLM-based robotics tasks. Our argument is simple: a USD is an XML-based scene graph, readable by LLMs and humans alike, and rich enough to support essentially any task -- Pixar developed this language to store assets, scenes and even movies. We demonstrate a ``Real to USD'' system using a Unitree Go2 quadruped robot carrying LiDAR and a RGB camera that (i) builds an explicit USD representation of indoor environments with diverse objects and challenging settings with lots of glass, and (ii) parses the USD using Google's Gemini to demonstrate scene understanding, complex inferences, and planning. We also study different aspects of this system in simulated warehouse and hospital settings using Nvidia's Issac Sim. Code is available at https://github.com/grasp-lyrl/Real2USD .

Figures (10)

• Figure 1: In simulation, we can use the underlying USD in Isaac Sim NVIDIA_Isaac_Sim along with a task prompt with an LLM to obtain a list of waypoints for a robot as it performs complex semantical tasks such as "Find all things that might be useful in case of an emergency". In the real world, we propose a framework called Real2USD (see \ref{['fig:real2usd']}) to build a faithful USD representation of the scene from sensor measurements (RGBD camera and pose estimates). Using this USD, a Unitree Go2 robot can perform semantic tasks in challenging indoor environments.
• Figure 2: Snapshots of the Spot robot at different waypoints in the hospital scene. The 3D mesh was built from simulated RGB, depth, and pose data using Khronos Schmid2024Khronos to demonstrate coverage.
• Figure 3: Inset images show safety equipment such as fire extinguishers and wet floor signs that were discovered by a Unitree H1 robot exploring the warehouse scene.
• Figure 4: We show the Real2USD pipeline where a robot collects sensor measurements and does semantic segmentation. With each image crop, it retrieves the most similar asset from our database, simulates the asset and does registration to a global point cloud. After some reconciliation, we obtain a USD in Isaac Sim that is fed to an LLM to get waypoints such that a robot can navigate to a language guided goal.
• Figure 5: An example of inadequate precision and recall in open-set detection models such as YOLOE. In the left image, the couch is typically categorized as a "church bench" (dark blue). We use heuristics to map such fine-grained but semantically incorrect labels to more general labels, e.g., the "church bench" is modified to be a "chair". Notice that the round table and desks are not segmented in these images, even with prompts such as "table" or "desk". This inadequacy is problematic for robotics pipelines that have much more stringent requirements on their performance. We work around this by querying the model twice (see the text).