Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning

NVIDIA, :, Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, Lukasz Wawrzyniak, Milad Rakhsha, Alain Denzler, Eric Heiden, Ales Borovicka, Ossama Ahmed, Iretiayo Akinola, Abrar Anwar, Mark T. Carlson, Ji Yuan Feng, Animesh Garg, Renato Gasoto, Lionel Gulich, Yijie Guo, M. Gussert, Alex Hansen, Mihir Kulkarni, Chenran Li, Wei Liu, Viktor Makoviychuk, Grzegorz Malczyk, Hammad Mazhar, Masoud Moghani, Adithyavairavan Murali, Michael Noseworthy, Alexander Poddubny, Nathan Ratliff, Welf Rehberg, Clemens Schwarke, Ritvik Singh, James Latham Smith, Bingjie Tang, Ruchik Thaker, Matthew Trepte, Karl Van Wyk, Fangzhou Yu, Alex Millane, Vikram Ramasamy, Remo Steiner, Sangeeta Subramanian, Clemens Volk, CY Chen, Neel Jawale, Ashwin Varghese Kuruttukulam, Michael A. Lin, Ajay Mandlekar, Karsten Patzwaldt, John Welsh, Huihua Zhao, Fatima Anes, Jean-Francois Lafleche, Nicolas Moënne-Loccoz, Soowan Park, Rob Stepinski, Dirk Van Gelder, Chris Amevor, Jan Carius, Jumyung Chang, Anka He Chen, Pablo de Heras Ciechomski, Gilles Daviet, Mohammad Mohajerani, Julia von Muralt, Viktor Reutskyy, Michael Sauter, Simon Schirm, Eric L. Shi, Pierre Terdiman, Kenny Vilella, Tobias Widmer, Gordon Yeoman, Tiffany Chen, Sergey Grizan, Cathy Li, Lotus Li, Connor Smith, Rafael Wiltz, Kostas Alexis, Yan Chang, David Chu, Linxi "Jim" Fan, Farbod Farshidian, Ankur Handa, Spencer Huang, Marco Hutter, Yashraj Narang, Soha Pouya, Shiwei Sheng, Yuke Zhu, Miles Macklin, Adam Moravanszky, Philipp Reist, Yunrong Guo, David Hoeller, Gavriel State

TL;DR

Isaac Lab tackles the challenge of scalable, multi-modal robot learning by delivering a GPU-native simulation platform built on USD/OpenUSD, PhysX, and RTX rendering. It unifies assets, actuators, sensors, teleoperation, and environment generation into modular workflows (manager-based and direct) that support RL, imitation learning, and synthetic data generation at scale. The framework demonstrates high-throughput simulation, diverse task suites, and advanced perception and control capabilities, with extensive benchmarks and a clear path toward integrating the Newton differentiable physics engine. This combination enables rapid prototyping, robust sim-to-real transfer, and the development of general-purpose robotics policies and foundation models. Looking forward, Newton integration and policy evaluation benchmarks position Isaac Lab as a central platform for scalable, real-world robotics research and deployment.

Abstract

We present Isaac Lab, the natural successor to Isaac Gym, which extends the paradigm of GPU-native robotics simulation into the era of large-scale multi-modal learning. Isaac Lab combines high-fidelity GPU parallel physics, photorealistic rendering, and a modular, composable architecture for designing environments and training robot policies. Beyond physics and rendering, the framework integrates actuator models, multi-frequency sensor simulation, data collection pipelines, and domain randomization tools, unifying best practices for reinforcement and imitation learning at scale within a single extensible platform. We highlight its application to a diverse set of challenges, including whole-body control, cross-embodiment mobility, contact-rich and dexterous manipulation, and the integration of human demonstrations for skill acquisition. Finally, we discuss upcoming integration with the differentiable, GPU-accelerated Newton physics engine, which promises new opportunities for scalable, data-efficient, and gradient-based approaches to robot learning. We believe Isaac Lab's combination of advanced simulation capabilities, rich sensing, and data-center scale execution will help unlock the next generation of breakthroughs in robotics research.

Isaac Lab: A GPU-Accelerated Simulation Framework for Multi-Modal Robot Learning

TL;DR

Isaac Lab tackles the challenge of scalable, multi-modal robot learning by delivering a GPU-native simulation platform built on USD/OpenUSD, PhysX, and RTX rendering. It unifies assets, actuators, sensors, teleoperation, and environment generation into modular workflows (manager-based and direct) that support RL, imitation learning, and synthetic data generation at scale. The framework demonstrates high-throughput simulation, diverse task suites, and advanced perception and control capabilities, with extensive benchmarks and a clear path toward integrating the Newton differentiable physics engine. This combination enables rapid prototyping, robust sim-to-real transfer, and the development of general-purpose robotics policies and foundation models. Looking forward, Newton integration and policy evaluation benchmarks position Isaac Lab as a central platform for scalable, real-world robotics research and deployment.

Abstract

We present Isaac Lab, the natural successor to Isaac Gym, which extends the paradigm of GPU-native robotics simulation into the era of large-scale multi-modal learning. Isaac Lab combines high-fidelity GPU parallel physics, photorealistic rendering, and a modular, composable architecture for designing environments and training robot policies. Beyond physics and rendering, the framework integrates actuator models, multi-frequency sensor simulation, data collection pipelines, and domain randomization tools, unifying best practices for reinforcement and imitation learning at scale within a single extensible platform. We highlight its application to a diverse set of challenges, including whole-body control, cross-embodiment mobility, contact-rich and dexterous manipulation, and the integration of human demonstrations for skill acquisition. Finally, we discuss upcoming integration with the differentiable, GPU-accelerated Newton physics engine, which promises new opportunities for scalable, data-efficient, and gradient-based approaches to robot learning. We believe Isaac Lab's combination of advanced simulation capabilities, rich sensing, and data-center scale execution will help unlock the next generation of breakthroughs in robotics research.

Paper Structure

This paper contains 70 sections, 2 equations, 35 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Core Simulation Infrastructure
  3. USD for Robotics
  4. Physics Simulation
  5. Rendering
  6. Isaac Lab Design and Features
  7. Assets
  8. Actuators
  9. Implicit Actuators
  10. Explicit Actuators
  11. Sensors
  12. Physics-based Sensors
  13. Rendering-based Camera
  14. Warp-based Ray-Caster
  15. Visuo-Tactile Sensor
  16. ...and 55 more sections

Figures (35)

  • Figure 1: Isaac Lab supports diverse robotic applications with exteroceptive observation inputs. It provides a user-friendly API for experimentation and includes features to facilitate sim-to-real transfer. The framework also supports multiple learning paradigms, including reinforcement learning and imitation learning.
  • Figure 2: Isaac Lab uses OpenUSD to define rich, complex simulation scenes for robotics. Robots, objects, and sensors are arranged in hierarchical scene graphs, where parent–child relationships manage spatial organization, coordinate frames, and groupings. Meanwhile, simulation-specific schemas capture visual appearance, collision geometry, physical properties, semantic IDs, and sensor configurations.
  • Figure 3: Integration of USD with PhysX in OmniPhysics. The USD stage provides a hierarchical representation of all objects and robots in the scene. This scene graph is parsed into PhysX, which allocates GPU tensors representing the internal simulation state. Unlike Isaac Gym, where users accessed raw buffers and manually indexed per simulation object, OmniPhysics exposes these states through the View APIs. These APIs allow read or write access to subsets of objects in the scene while simplifying data management and improving usability.
  • Figure 4: Photo-realistic rendering in Isaac Lab using the Omniverse RTX renderer, demonstrating high-quality ray tracing with complex physically-based materials authored using NVIDIA’s MDL. The rendering showcases realistic effects such as reflections and refractions, resulting in visually rich and high-quality scenes.
  • Figure 5: Tiled rendering of multiple simulated environments. Each environment has a separate camera, and their outputs are spatially tiled into a single GPU frame-buffer. The deterministic layout allows efficient reconstruction of per-environment observations without costly host–device transfers.
  • ...and 30 more figures