A Reinforcement Learning Based Controller to Minimize Forces on the Crutches of a Lower-Limb Exoskeleton
Aydin Emre Utku, Suzan Ece Ada, Muhammet Hatipoglu, Mustafa Derman, Emre Ugur, Evren Samur
TL;DR
This work tackles the problem of high upper-body energy expenditure in powered lower-limb exoskeletons by minimizing ground reaction forces on crutches through a PPO-based deep RL controller trained in a MuJoCo real-to-sim framework. By designing a multi-term reward that explicitly penalizes crutch GRF while enforcing forward progression and stability, the authors enable the policy to generate joint torques directly from state observations and measured GRFs. The results show that incorporating a crutch force penalty yields substantial reductions in crutch GRF (up to 35% relative to a baseline) and produces stable, forward walking patterns, though excessively large GRF weights can destabilize learning. Overall, the study demonstrates the feasibility of RL-based control to reduce upper-body effort in exoskeleton-assisted gait and highlights the value of reward shaping for ergonomic improvements in assistive devices.
Abstract
Metabolic energy consumption of a powered lower-limb exoskeleton user mainly comes from the upper body effort since the lower body is considered to be passive. However, the upper body effort of the users is largely ignored in the literature when designing motion controllers. In this work, we use deep reinforcement learning to develop a locomotion controller that minimizes ground reaction forces (GRF) on crutches. The rationale for minimizing GRF is to reduce the upper body effort of the user. Accordingly, we design a model and a learning framework for a human-exoskeleton system with crutches. We formulate a reward function to encourage the forward displacement of a human-exoskeleton system while satisfying the predetermined constraints of a physical robot. We evaluate our new framework using Proximal Policy Optimization, a state-of-the-art deep reinforcement learning (RL) method, on the MuJoCo physics simulator with different hyperparameters and network architectures over multiple trials. We empirically show that our learning model can generate joint torques based on the joint angle, velocities, and the GRF on the feet and crutch tips. The resulting exoskeleton model can directly generate joint torques from states in line with the RL framework. Finally, we empirically show that policy trained using our method can generate a gait with a 35% reduction in GRF with respect to the baseline.