Scenario Execution for Robotics: A generic, backend-agnostic library for running reproducible robotics experiments and tests

TL;DR

This paper proposes a novel backend- and middleware-agnostic approach for conducting systematic, reproducible and automatable robotics experiments called Scenario Execution for Robotics, implemented as a Python library built on top of the generic scenario description language OpenSCENARIO 2 and Behavior Trees.

Abstract

Testing and evaluation of robotics systems is a difficult and oftentimes tedious task due to the systems' complexity and a lack of tools to conduct reproducible robotics experiments. Additionally, almost all available tools are either tailored towards a specific application domain, simulator or middleware. Particularly scenario-based testing, a common practice in the domain of automated driving, is not sufficiently covered in the robotics domain. In this paper, we propose a novel backend- and middleware-agnostic approach for conducting systematic, reproducible and automatable robotics experiments called Scenario Execution for Robotics. Our approach is implemented as a Python library built on top of the generic scenario description language OpenSCENARIO 2 and Behavior Trees and is made publicly available on GitHub. In extensive experiments, we demonstrate that our approach supports multiple simulators as backend and can be used as a standalone Python-library or as part of the ROS2 ecosystem. Furthermore, we demonstrate how our approach enables testing over ranges of varying values. Finally, we show how Scenario Execution for Robotics allows to move from simulation-based to real-world experiments with minimal adaptations to the scenario description file.

Figures (7)

• Figure 1: High-level overview of the SERo library.
• Figure 2: Overview of the modular software architecture of the SERo library.
• Figure 3: Snapshot of scenario execution realizing a simulated navigation scenario with the Turtlebot4 simulation in Gazebo and ROS2.
• Figure 4: Several selected snapshots of a pick-and-place scenario in the PyBullet simulator realized with SERo. From left to right: a) after spawning all actors and objects, b) while moving the end-effector to the grasping position, c) reached grasping position, now move (i.e., close) gripper, d) move end-effector up to reach successful final pick-up pose.
• Figure 5: Snapshot of selected runs of the abstract parameter variation scenario described in Sec. \ref{['ssec:varying_params_and_faults']} with different values for the standard deviation (namely 0.1, 0.2, 0.5 from left to right) of the Gaussian noise added to the LIDAR scan