Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotion

TL;DR

This work advances musculoskeletal locomotion control by applying adversarial imitation learning to a 16-DOF humanoid actuated by 92 MTUs, learning natural walking and running gaits from a small set of demonstrations. It systematically analyzes policy distributions, exploration objectives, and muscle synergies to address the exploration challenges inherent in over-actuated systems, introducing Latent Exploration and Synergistic Action Representation as key approaches. The findings show that direct mean regularization (via flipped uniform KL or out-of-bounds penalties) and synergy-aware exploration substantially improve asymptotic performance, with Latent Exploration plus out-of-bounds penalty delivering near-optimal walking and competitive running. This work lays groundwork for biomechanical realism in gait policies and informs gait assistive device design through principled imitation learning in high-dimensional muscle-actuated systems.

Abstract

Learning a locomotion controller for a musculoskeletal system is challenging due to over-actuation and high-dimensional action space. While many reinforcement learning methods attempt to address this issue, they often struggle to learn human-like gaits because of the complexity involved in engineering an effective reward function. In this paper, we demonstrate that adversarial imitation learning can address this issue by analyzing key problems and providing solutions using both current literature and novel techniques. We validate our methodology by learning walking and running gaits on a simulated humanoid model with 16 degrees of freedom and 92 Muscle-Tendon Units, achieving natural-looking gaits with only a few demonstrations.

Figures (6)

• Figure 1: Humanoid model with 16 DOFs actuated by 92 Muscle-Tendon Units during running (left) and walking (right).
• Figure 2: Basic composition of a Muscle-Tendon Unit
• Figure 3: Comparison of exploration objectives depending on the mean of the gaussian.
• Figure 4: Target velocity performance curves for imitation learning of walking and running with the different (a) policy distributions (b) exploration objectives and (c) methods for synergistic exploration introduced in section \ref{['sec:method']}. All experiments were started with 15 seeds.
• Figure 5: Development of the policy distribution's entropy and absolute action mean under the different exploration objective setups.