Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approach

Ammar N. Abbas, Shakra Mehak, Georgios C. Chasparis, John D. Kelleher, Michael Guilfoyle, Maria Chiara Leva, Aswin K Ramasubramanian

TL;DR

This work tackles safety concerns in deep reinforcement learning for collaborative robots by embedding ISO 10218 safety constraints and IEC 61508 functional-safety evaluation into a Sim2Real DRL framework. The authors introduce Safety-Driven DRL (SD-DRL) with a safety-enhanced reward that penalizes speed, collisions, and violation of safety limits, and validate it on a UR5 grasping task using UR5GraspEnv-v0 across simulation and real hardware. Key contributions include the modified reward function, a thorough safety and SIL-based validation, and demonstrations of improved safety-compliant performance (SIL 2) while maintaining efficiency. The approach demonstrates the viability of deploying DRL in safety-critical cobot applications and provides a path toward standardized safety verification in learning-enabled robotic systems.

Abstract

This study presents a novel methodology incorporating safety constraints into a robotic simulation during the training of deep reinforcement learning (DRL). The framework integrates specific parts of the safety requirements, such as velocity constraints, as specified by ISO 10218, directly within the DRL model that becomes a part of the robot's learning algorithm. The study then evaluated the efficiency of these safety constraints by subjecting the DRL model to various scenarios, including grasping tasks with and without obstacle avoidance. The validation process involved comprehensive simulation-based testing of the DRL model's responses to potential hazards and its compliance. Also, the performance of the system is carried out by the functional safety standards IEC 61508 to determine the safety integrity level. The study indicated a significant improvement in the safety performance of the robotic system. The proposed DRL model anticipates and mitigates hazards while maintaining operational efficiency. This study was validated in a testbed with a collaborative robotic arm with safety sensors and assessed with metrics such as the average number of safety violations, obstacle avoidance, and the number of successful grasps. The proposed approach outperforms the conventional method by a 16.5% average success rate on the tested scenarios in the simulations and 2.5% in the testbed without safety violations. The project repository is available at https://github.com/ammar-n-abbas/sim2real-ur-gym-gazebo.

Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approach

TL;DR

This work tackles safety concerns in deep reinforcement learning for collaborative robots by embedding ISO 10218 safety constraints and IEC 61508 functional-safety evaluation into a Sim2Real DRL framework. The authors introduce Safety-Driven DRL (SD-DRL) with a safety-enhanced reward that penalizes speed, collisions, and violation of safety limits, and validate it on a UR5 grasping task using UR5GraspEnv-v0 across simulation and real hardware. Key contributions include the modified reward function, a thorough safety and SIL-based validation, and demonstrations of improved safety-compliant performance (SIL 2) while maintaining efficiency. The approach demonstrates the viability of deploying DRL in safety-critical cobot applications and provides a path toward standardized safety verification in learning-enabled robotic systems.

Abstract

This study presents a novel methodology incorporating safety constraints into a robotic simulation during the training of deep reinforcement learning (DRL). The framework integrates specific parts of the safety requirements, such as velocity constraints, as specified by ISO 10218, directly within the DRL model that becomes a part of the robot's learning algorithm. The study then evaluated the efficiency of these safety constraints by subjecting the DRL model to various scenarios, including grasping tasks with and without obstacle avoidance. The validation process involved comprehensive simulation-based testing of the DRL model's responses to potential hazards and its compliance. Also, the performance of the system is carried out by the functional safety standards IEC 61508 to determine the safety integrity level. The study indicated a significant improvement in the safety performance of the robotic system. The proposed DRL model anticipates and mitigates hazards while maintaining operational efficiency. This study was validated in a testbed with a collaborative robotic arm with safety sensors and assessed with metrics such as the average number of safety violations, obstacle avoidance, and the number of successful grasps. The proposed approach outperforms the conventional method by a 16.5% average success rate on the tested scenarios in the simulations and 2.5% in the testbed without safety violations. The project repository is available at https://github.com/ammar-n-abbas/sim2real-ur-gym-gazebo.
Paper Structure (27 sections, 4 equations, 4 figures, 4 tables)

This paper contains 27 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. State-of-the-Art
  3. Methodology
  4. Environment Framework
  5. Deep Reinforcement Learning
  6. Algorithm and Hyperparameters
  7. State
  8. Action
  9. Reward
  10. Evaluation Metrics
  11. Average Episode Returns
  12. Average Number of Violations per Episode
  13. Success Rate
  14. Safety-Driven Success Rate
  15. Safety Standard Compliance
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Safety-Driven Deep Reinforcement Learning.
  • Figure 2: Sim2Real environment framework.
  • Figure 3: Policy testing for (a) grasping on simulation, (b) grasping on the testbed, (c) and (d) grasping with obstacle avoidance on the testbed.
  • Figure 4: Velocity profiles for (a) DRL and (b) Safety-Driven DRL during a collision.