Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments

TL;DR

Orbit provides a fast, realistic, and extensible robot-learning simulation platform by unifying rigid and deformable physics within NVIDIA Isaac Sim and PhysX 5. It offers a rich collection of robots, sensors, motion generators, and tasks, plus wrappers for multiple RL and IL frameworks, enabling RL, imitation learning, and motion planning at scale. The framework emphasizes modular abstractions (World, Agent, and learning tasks) and demonstrates strong throughput and sim-to-real potential through exemplar workflows in RL, IL, and motion planning. By combining high-fidelity simulation, versatile scene design, and broad task coverage, Orbit aims to lower barriers to entry and catalyze interdisciplinary robotics research.

Abstract

We present Orbit, a unified and modular framework for robot learning powered by NVIDIA Isaac Sim. It offers a modular design to easily and efficiently create robotic environments with photo-realistic scenes and high-fidelity rigid and deformable body simulation. With Orbit, we provide a suite of benchmark tasks of varying difficulty -- from single-stage cabinet opening and cloth folding to multi-stage tasks such as room reorganization. To support working with diverse observations and action spaces, we include fixed-arm and mobile manipulators with different physically-based sensors and motion generators. Orbit allows training reinforcement learning policies and collecting large demonstration datasets from hand-crafted or expert solutions in a matter of minutes by leveraging GPU-based parallelization. In summary, we offer an open-sourced framework that readily comes with 16 robotic platforms, 4 sensor modalities, 10 motion generators, more than 20 benchmark tasks, and wrappers to 4 learning libraries. With this framework, we aim to support various research areas, including representation learning, reinforcement learning, imitation learning, and task and motion planning. We hope it helps establish interdisciplinary collaborations in these communities, and its modularity makes it easily extensible for more tasks and applications in the future.

Figures (6)

• Figure 1: Orbit's abstractions comprise World, analogous to the real world, and Agent, the computation graph behind the embodied system. The nodes in the agent's graph can perform observation-based or action-based processing. Through a graph-cut over this computation graph and specifying an extrinsic goal, it is feasible to design different tasks within the same World definition. For instance, to perform RL, this would define the task with $o_t$, $a_t$, and $r_t$ corresponding to the observation, action, and reward signals respectively.
• Figure 2: Illustration of actuator groups for a legged mobile manipulator. This allows decomposing a complex system into sub-groups and defining of specific transmission models for each of them flexibly. The number inside $(\cdot)$ is the dimension of the command vector.
• Figure 3: Overview of features included in Orbit. We provide models of different sensors, robotic platforms, objects from different datasets, motion generators, and teleoperation devices. Using RTX-accelerated ray-tracing, we can obtain high-fidelity images in real-time for different modalities such as RGB, depth, surface normal, instance, and semantic segmentation (pixel-wise and bounding boxes).
• Figure 4: Demonstration of the designed tasks using hand-crafted state machines and task-space controllers. Leveraging recent advances in physics engines, we support high-fidelity simulation of rigid and deformable objects. We include environments that allow switching between robots, objects, observations, and action spaces through configuration files (https://isaac-orbit.github.io/#SampleTasks).
• Figure 5: Example showing RL integration. We include wrappers to various RL frameworks. Additionally, it is possible to easily switch action spaces for training policies with different controllers. The plot shows the mean of the average return over five seeds.