An Open-source Sim2Real Approach for Sensor-independent Robot Navigation in a Grid

TL;DR

This work addresses the Sim2Real gap by bootstrapping policy transfer from a grid-based simulation to a real quadruped without sensors. It trains a Q-learning agent in the Gymnasium Frozen Lake environment, derives a Q-table, and converts it into a header-driven motion sequence to drive a 12-DOF quadruped via Arduino, enabling sensor-free grid navigation. Key contributions include an open-source, low-cost, end-to-end pipeline, a 4×4 grid demonstration with obstacle avoidance, and a demonstrated generalization to different grid scales through a movement multiplier. The approach offers an educational and hobbyist-friendly framework for RL-based robot navigation, expanding access to practical Sim2Real experimentation.

Abstract

This paper presents a Sim2Real (Simulation to Reality) approach to bridge the gap between a trained agent in a simulated environment and its real-world implementation in navigating a robot in a similar setting. Specifically, we focus on navigating a quadruped robot in a real-world grid-like environment inspired by the Gymnasium Frozen Lake -- a highly user-friendly and free Application Programming Interface (API) to develop and test Reinforcement Learning (RL) algorithms. We detail the development of a pipeline to transfer motion policies learned in the Frozen Lake simulation to a physical quadruped robot, thus enabling autonomous navigation and obstacle avoidance in a grid without relying on expensive localization and mapping sensors. The work involves training an RL agent in the Frozen Lake environment and utilizing the resulting Q-table to control a 12 Degrees-of-Freedom (DOF) quadruped robot. In addition to detailing the RL implementation, inverse kinematics-based quadruped gaits, and the transfer policy pipeline, we open-source the project on GitHub and include a demonstration video of our Sim2Real transfer approach. This work provides an accessible, straightforward, and low-cost framework for researchers, students, and hobbyists to explore and implement RL-based robot navigation in real-world grid environments.

Figures (6)

• Figure 1: Gymnasium Frozen Lake-inspired Sim2Real architecture for grid navigation
• Figure 2: Quadruped robot kinematics
• Figure 3: Assembled quadruped robot
• Figure 4: Performance of the agent over 500 episodes of training
• Figure 5: The Sim2Real Transfer Experiment: Demonstration of a quadruped robot navigating in a real grid of size 182 $\times$ 182 cm using a policy learned in Frozen Lake simulation. Four obstacles placed in the grid are customized accordingly in the Frozen Lake simulation. The robot autonomously avoids obstacles without any sensor feedback and reaches the destination using the shortest path.