Investigating Adaptive Tuning of Assistive Exoskeletons Using Offline Reinforcement Learning: Challenges and Insights

TL;DR

The paper tackles automatic tuning of upper-limb exoskeleton parameters using offline multi-agent reinforcement learning (MARL). It frames two agents to optimize continuous biceps and triceps effort thresholds via Mixed Q-Functionals (MQF) to leverage pre-collected data without online exploration. Experiments on the MyoPro 2 across horizontal and vertical tasks indicate the approach can dynamically adjust thresholds to improve control, though evaluation remains limited by dataset size and coverage. This work demonstrates a data-efficient pathway toward personalized, safer exoskeleton control, with future work expanding datasets and conducting real-world testing.

Abstract

Assistive exoskeletons have shown great potential in enhancing mobility for individuals with motor impairments, yet their effectiveness relies on precise parameter tuning for personalized assistance. In this study, we investigate the potential of offline reinforcement learning for optimizing effort thresholds in upper-limb assistive exoskeletons, aiming to reduce reliance on manual calibration. Specifically, we frame the problem as a multi-agent system where separate agents optimize biceps and triceps effort thresholds, enabling a more adaptive and data-driven approach to exoskeleton control. Mixed Q-Functionals (MQF) is employed to efficiently handle continuous action spaces while leveraging pre-collected data, thereby mitigating the risks associated with real-time exploration. Experiments were conducted using the MyoPro 2 exoskeleton across two distinct tasks involving horizontal and vertical arm movements. Our results indicate that the proposed approach can dynamically adjust threshold values based on learned patterns, potentially improving user interaction and control, though performance evaluation remains challenging due to dataset limitations.

Figures (7)

• Figure 1: Overview of the MyoPro 2 device, highlighting the placement of sEMG sensors.
• Figure 2: Illustration of joint speed as a function of delta effort, highlighting the relationship between biceps and triceps activation. The graph is divided into three regions: flex, where triceps activation exceeds the threshold, extend, where biceps activation surpasses the threshold, and idle, where neither muscle reaches activation. The placement of triceps and biceps thresholds is marked, demonstrating their influence on movement transitions.
• Figure 3: Visualization of muscle activation patterns, corresponding effort levels, and resulting actions. The left column categorizes different activation scenarios, including biceps contraction, triceps contraction, co-contraction, and imbalanced muscle activation. The middle column represents effort levels using effort bars, where biceps and triceps contributions are indicated. The right column shows the resulting action, such as pure flexion, pure extension, no movement, or imbalanced extensions and flexions, based on the activation pattern.
• Figure 4: Overview of the proposed offline MARL architecture with MQF: (a) training prediction blocks, (b) updating target blocks. The red arrows indicate the direction of the backpropagation, while the blue arrows depict the target network updates for mixer and learners' prediction blocks.
• Figure 5: Demonstration of task movements for data collection using the MyoPro exoskeleton. (a) Depicts the horizontal task, where the user moves their arm laterally. (b) Shows the vertical task, involving an upward and downward arm motion.