EAGERx: Graph-Based Framework for Sim2real Robot Learning

TL;DR

EAGERx tackles the sim2real gap by introducing a graph-based, engine-agnostic framework that enables synchronized, faster-than-real-time simulation with delay modeling and domain randomization. It defines a modular architecture (Graph/Node/Object/Engine/Backend/BaseEnv) and a novel synchronization protocol that coordinates asynchronous components to maintain consistent behavior across simulators and real hardware. Key contributions include the synchronization protocol, engine-agnostic design, modular reset and simulator augmentation capabilities, and demonstrated improvements in zero-shot sim2real transfer across pendulum, vision-based manipulation, and dynamic quadrotor tasks. The approach offers practical impact by enabling flexible multi-engine pipelines, interactive learning workflows, and ML-assisted control, all under an open-source implementation with strong documentation and tutorials.

Abstract

Sim2real, that is, the transfer of learned control policies from simulation to real world, is an area of growing interest in robotics due to its potential to efficiently handle complex tasks. The sim2real approach faces challenges due to mismatches between simulation and reality. These discrepancies arise from inaccuracies in modeling physical phenomena and asynchronous control, among other factors. To this end, we introduce EAGERx, a framework with a unified software pipeline for both real and simulated robot learning. It can support various simulators and aids in integrating state, action and time-scale abstractions to facilitate learning. EAGERx's integrated delay simulation, domain randomization features, and proposed synchronization algorithm contribute to narrowing the sim2real gap. We demonstrate (in the context of robot learning and beyond) the efficacy of EAGERx in accommodating diverse robotic systems and maintaining consistent simulation behavior. EAGERx is open source and its code is available at https://eagerx.readthedocs.io.

Figures (12)

• Figure 1: Our framework offers a unified software pipeline for both simulated and real robot learning. It can support various simulators and aids in integrating state, action and time-scale abstractions.
• Figure 2: (a) Displays the engine-agnostic graph of the pendulum environment from Code-Example \ref{['code:main']} as generated by the GUI. The engine-specific subgraphs for replacing the object (i.e., pendulum) are depicted for the ODE (b) and real-world (c) engines. The yellow nodes, split for visualization clarity, symbolize the agent's actions and observations. Blue squares represent I/O channels, while red squares indicate node states and/or parameters that can be randomized at the start of an episode.
• Figure 3: Diverse robotic system tasks illustrating the EAGERx framework's flexibility. (a) Swing-up task with an inverted pendulum, highlighting delay compensation in reinforcement learning. The task involves zero-shot evaluations on a real-world pendulum setup, comparing a disk-based simulator with the OpenAI Gym rod-based environment. (b) Box-pushing experiment using a Viper 300x robotic manipulator, emphasizing the need for domain randomization with a low-resolution Logitech C170 webcam for box localization tracking. (c) Inclined landing task where a quadrotor lands on a moving and inclined deck, showcasing the integration of multiple mobile robots into a dynamic task.
• Figure 4: Comparison of mean episodic cost between simulations and real-world pendulum performance. The success threshold denotes the level below which a 100% success rate is achieved. Performance drops notably in the real-world scenario with a conventional gym approach, illustrating the sim2real gap. Asynchronous simulation (async) at real-time speeds mitigates the gap but leads to excessively long training times. Synchronized training under our protocol (EAGERx) facilitates consistent performance at faster-than-real-time simulation speeds.
• Figure 5: The impact of varying real-time factors (rtf) on the mean episodic cost in a simulated pendulum environment. Performance declines as the rtf increases, indicating the challenges of maintaining fidelity in faster-than-real-time simulations when components operate asynchronously.