Towards Building AI-CPS with NVIDIA Isaac Sim: An Industrial Benchmark and Case Study for Robotics Manipulation

TL;DR

This paper builds a public, industrial-grade benchmark for AI-enabled robotics manipulation on NVIDIA Omniverse Isaac Sim, featuring eight manipulation tasks and multiple DRL controllers to study performance and reliability. It also introduces a Python-based falsification framework compatible with physics simulators and OpenAI Gym environments to assess AI controllers against STL specifications, using optimizers such as random, Nelder-Mead, and dual annealing. Experimental results show DRL controllers (notably TRPO and PPO) achieving high success rates and robustness to action noise, while falsification reveals reliability gaps beyond reward-based evaluation. Overall, the work provides a practical, extensible pipeline for designing, evaluating, and testing AI-enabled robotics CPS, bridging simulation with real-world considerations and laying groundwork for trustworthy industrial AI robotics.

Abstract

As a representative cyber-physical system (CPS), robotic manipulator has been widely adopted in various academic research and industrial processes, indicating its potential to act as a universal interface between the cyber and the physical worlds. Recent studies in robotics manipulation have started employing artificial intelligence (AI) approaches as controllers to achieve better adaptability and performance. However, the inherent challenge of explaining AI components introduces uncertainty and unreliability to these AI-enabled robotics systems, necessitating a reliable development platform for system design and performance assessment. As a foundational step towards building reliable AI-enabled robotics systems, we propose a public industrial benchmark for robotics manipulation in this paper. It leverages NVIDIA Omniverse Isaac Sim as the simulation platform, encompassing eight representative manipulation tasks and multiple AI software controllers. An extensive evaluation is conducted to analyze the performance of AI controllers in solving robotics manipulation tasks, enabling a thorough understanding of their effectiveness. To further demonstrate the applicability of our benchmark, we develop a falsification framework that is compatible with physical simulators and OpenAI Gym environments. This framework bridges the gap between traditional testing methods and modern physics engine-based simulations. The effectiveness of different optimization methods in falsifying AI-enabled robotics manipulation with physical simulators is examined via a falsification test. Our work not only establishes a foundation for the design and development of AI-enabled robotics systems but also provides practical experience and guidance to practitioners in this field, promoting further research in this critical academic and industrial domain.

Figures (6)

• Figure 1: Workflow summary of this paper.
• Figure 2: Abstract workflow of CPS
• Figure 3: RQ1 - Partial questions and results from survey
• Figure 4: Robotics manipulation tasks included in the proposed benchmark based on Isaac Sim.
• Figure 5: Parallel running of environments in Isaac Sim.