Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Closing the Sim2Real Performance Gap in RL

Akhil S Anand, Shambhuraj Sawant, Jasper Hoffmann, Dirk Reinhardt, Sebastien Gros

TL;DR

This paper tackles the persistent Sim2Real performance gap by reframing simulator adaptation as a bi-level reinforcement learning problem. The inner loop trains an in-sim policy using stochastic policy gradient (SPG) within a parameterized simulator, while the outer loop updates the simulator parameters to maximize real-world policy performance, with the two loops linked through an implicit function condition. The authors derive sensitivity tools based on implicit differentiation to compute the outer-loop gradients, and present two illustrative examples (discrete and continuous MDPs) to validate the approach. They argue that this objective-oriented, differentiable-planning perspective can close the Sim2Real gap more effectively than proxy-based methods, albeit with significant computational considerations and several practical limitations to address in future work. The work lays a theoretical foundation for integrating differentiable simulators and outer-loop optimization into Model-Based RL for real-world deployment and suggests directions for scalable algorithmic development and offline outer-loop training.

Abstract

Sim2Real aims at training policies in high-fidelity simulation environments and effectively transferring them to the real world. Despite the developments of accurate simulators and Sim2Real RL approaches, the policies trained purely in simulation often suffer significant performance drops when deployed in real environments. This drop is referred to as the Sim2Real performance gap. Current Sim2Real RL methods optimize the simulator accuracy and variability as proxies for real-world performance. However, these metrics do not necessarily correlate with the real-world performance of the policy as established theoretically and empirically in the literature. We propose a novel framework to address this issue by directly adapting the simulator parameters based on real-world performance. We frame this problem as a bi-level RL framework: the inner-level RL trains a policy purely in simulation, and the outer-level RL adapts the simulation model and in-sim reward parameters to maximize real-world performance of the in-sim policy. We derive and validate in simple examples the mathematical tools needed to develop bi-level RL algorithms that close the Sim2Real performance gap.

Closing the Sim2Real Performance Gap in RL

TL;DR

This paper tackles the persistent Sim2Real performance gap by reframing simulator adaptation as a bi-level reinforcement learning problem. The inner loop trains an in-sim policy using stochastic policy gradient (SPG) within a parameterized simulator, while the outer loop updates the simulator parameters to maximize real-world policy performance, with the two loops linked through an implicit function condition. The authors derive sensitivity tools based on implicit differentiation to compute the outer-loop gradients, and present two illustrative examples (discrete and continuous MDPs) to validate the approach. They argue that this objective-oriented, differentiable-planning perspective can close the Sim2Real gap more effectively than proxy-based methods, albeit with significant computational considerations and several practical limitations to address in future work. The work lays a theoretical foundation for integrating differentiable simulators and outer-loop optimization into Model-Based RL for real-world deployment and suggests directions for scalable algorithmic development and offline outer-loop training.

Abstract

Sim2Real aims at training policies in high-fidelity simulation environments and effectively transferring them to the real world. Despite the developments of accurate simulators and Sim2Real RL approaches, the policies trained purely in simulation often suffer significant performance drops when deployed in real environments. This drop is referred to as the Sim2Real performance gap. Current Sim2Real RL methods optimize the simulator accuracy and variability as proxies for real-world performance. However, these metrics do not necessarily correlate with the real-world performance of the policy as established theoretically and empirically in the literature. We propose a novel framework to address this issue by directly adapting the simulator parameters based on real-world performance. We frame this problem as a bi-level RL framework: the inner-level RL trains a policy purely in simulation, and the outer-level RL adapts the simulation model and in-sim reward parameters to maximize real-world performance of the in-sim policy. We derive and validate in simple examples the mathematical tools needed to develop bi-level RL algorithms that close the Sim2Real performance gap.
Paper Structure (21 sections, 3 theorems, 63 equations, 3 figures, 1 algorithm)

This paper contains 21 sections, 3 theorems, 63 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sim2Real RL Approaches:
  4. Objective Mismatch in mbrl:
  5. Preliminaries
  6. Reinforcement Learning and Optimal Decision-Making
  7. In-sim Reinforcement Learning
  8. Optimal Sim2Real RL
  9. Stochastic Policy Gradient for Bi-level RL
  10. Illustrative Examples
  11. Discrete MDP:
  12. Continuous MDP:
  13. Discussions
  14. Limitations
  15. Conclusion
  16. ...and 6 more sections

Key Result

Lemma 1

Sensitivity of the expected value operation of a function $\eta(s_k,a_k)$ under a Markov chain induced by policy $\pi_\phi$ under the transition model $f_\theta$ is given by:

Figures (3)

  • Figure 1: (left) Classical Sim2Real rl, (right) the proposed bi-level rl framework.
  • Figure 2: Discrete MDP.
  • Figure 3: Normalized return of 5 randomized trials for (left) the discrete mdp example and (right) continuous mdp example.

Theorems & Definitions (3)

  • Lemma 1: Sensitivity of a Markov Chain
  • Lemma 2: Sensitivities of $Q^{\pi}$ and $V^{\pi}$
  • Theorem 1: Sensitivity of in-sim spg