Spectral Normalization for Lipschitz-Constrained Policies on Learning Humanoid Locomotion

TL;DR

This work tackles the sim-to-real transfer challenge for humanoid locomotion by addressing the tendency of RL policies to exploit unrealistically infinite actuator bandwidth. It introduces Spectral Normalization (SN) to enforce Lipschitz continuity in policy networks, providing finite-bandwidth action outputs without explicit gradient penalties. In simulation and on the humanoid TOCABI, SN achieves control stability comparable to gradient-penalty approaches while dramatically reducing GPU memory usage, enabling larger parallel training. Real-robot experiments confirm stable walking under SN across multiple command types, underscoring its practical relevance for robust, scalable locomotion learning.

Abstract

Reinforcement learning (RL) has shown great potential in training agile and adaptable controllers for legged robots, enabling them to learn complex locomotion behaviors directly from experience. However, policies trained in simulation often fail to transfer to real-world robots due to unrealistic assumptions such as infinite actuator bandwidth and the absence of torque limits. These conditions allow policies to rely on abrupt, high-frequency torque changes, which are infeasible for real actuators with finite bandwidth. Traditional methods address this issue by penalizing aggressive motions through regularization rewards, such as joint velocities, accelerations, and energy consumption, but they require extensive hyperparameter tuning. Alternatively, Lipschitz-Constrained Policies (LCP) enforce finite bandwidth action control by penalizing policy gradients, but their reliance on gradient calculations introduces significant GPU memory overhead. To overcome this limitation, this work proposes Spectral Normalization (SN) as an efficient replacement for enforcing Lipschitz continuity. By constraining the spectral norm of network weights, SN effectively limits high-frequency policy fluctuations while significantly reducing GPU memory usage. Experimental evaluations in both simulation and real-world humanoid robot show that SN achieves performance comparable to gradient penalty methods while enabling more efficient parallel training.

Figures (5)

• Figure 1: Comparison of policies trained using Baseline, Gradient Penalty-Based Lipschitz-Constrained Policy (GP-LCP), and Spectral Normalization (SN). The figure illustrates the regularization effects across methods. While the Baseline exhibits vibrations, GP-LCP stabilizes actions at the cost of increased computational overhead. SN achieves similar stability with lower computational overhead, demonstrating its effectiveness in enforcing finite bandwidth action control.
• Figure 2: Comparison between a standard actor network and an actor network with SN. In the SN-based network, each weight matrix $W_l$ is divided by its largest singular value $\sigma(W_l)$, effectively constraining the Lipschitz constant of the network. Unlike conventional networks, where weights can grow arbitrarily, SN ensures bounded transformations at each layer.
• Figure 3: Estimated squared gradient norm during training process
• Figure 4: Maximum GPU memory allocation during training across different methods (Baseline, Reg (Regularization Reward), GP-LCP, and SN (Ours)) and multiple random seeds (42, 777, 2025). The results show that GP-LCP requires significantly more memory due to gradient penalty computations, while SN achieves comparable performance with lower memory overhead.
• Figure 5: The plot compares training rewards per iteration for different numbers of environments (8192 vs. 4096) in simulation. The reduced GPU memory usage from SN, which eliminates the need for gradient penalty computations, allows for a higher number of parallel environments, leading to improved training efficiency.

Theorems & Definitions (1)

• Definition 1