Learning to Hop for a Single-Legged Robot with Parallel Mechanism

TL;DR

The paper tackles continuous hopping control for a highly dynamic single-legged robot using a parallel 3-RSR leg, where accurate simulation of the parallel mechanism is difficult and the aerial phase is prolonged. It proposes a reinforcement learning framework trained on a serial-template model with a torque-level conversion to bridge serial training and real parallel hardware, augmented by a beta-VAE encoder–decoder for long-history proprioception and explicit base-velocity estimation. The approach includes domain randomization and a Jacobian-based torque mapping to enable robust sim-to-real transfer, with ablation results showing the importance of explicit velocity and latent-state representations. Real-world experiments demonstrate zero-shot transfer and superior performance against a SLIP-based baseline, indicating practical impact for underactuated parallel-legged robots and suggesting broader applicability to similar systems.

Abstract

This work presents the application of reinforcement learning to improve the performance of a highly dynamic hopping system with a parallel mechanism. Unlike serial mechanisms, parallel mechanisms can not be accurately simulated due to the complexity of their kinematic constraints and closed-loop structures. Besides, learning to hop suffers from prolonged aerial phase and the sparse nature of the rewards. To address them, we propose a learning framework to encode long-history feedback to account for the under-actuation brought by the prolonged aerial phase. In the proposed framework, we also introduce a simplified serial configuration for the parallel design to avoid directly simulating parallel structure during the training. A torque-level conversion is designed to deal with the parallel-serial conversion to handle the sim-to-real issue. Simulation and hardware experiments have been conducted to validate this framework.

Figures (14)

• Figure 1: Overview of the hopping control framework for the single-legged robot with a parallel mechanism using reinforcement learning.
• Figure 2: Design and major parts of the 3-RSR single-legged hopping robot. The robot foot has three degrees of translational freedom.
• Figure 3: Hopping pattern of the 3D single-legged hopping robot.
• Figure 4: Actor policy structure of the proposed method. The encoder $\phi$ receives a history ($H=5$) of observation and uses a multi-head structure to output a latent vector $\boldsymbol{\mu}_t$, and an estimation $(\dot{\mathbf{q}}_{[x,y,z],t},\mathbf{c}_t))$. The latent vector is then passed through the decoder $\beta$ to reconstruct a future observation $\mathbf{o}_{t+1}$. Following the design of $\beta$-VAE higgins2017beta, a reconstruction loss and a KL-divergence regulation loss are used to train the encoder-decoder. The actor receives the latent vector $\boldsymbol{\mu}_t$, current observation $\mathbf{o}_t$ and the estimated state $(\dot{\mathbf{q}}_{[x,y,z],t})$ to produce an action $\mathbf{a}_t$. A torque level conversion described in Eq. \ref{['Eq_torque_conversion']} is done to map the serial joint torque to the parallel joint position.
• Figure 5: Training scenarios in the simulation for the single-legged robot. Four types of terrains including slopes and stairs are randomized and set up in the simulation.