Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-world Reinforcement Learning from Suboptimal Interventions

Yinuo Zhao, Huiqian Jin, Lechun Jiang, Xinyi Zhang, Kun Wu, Pei Ren, Zhiyuan Xu, Zhengping Che, Lei Sun, Dapeng Wu, Chi Harold Liu, Jian Tang

TL;DR

This paper addresses real-world RL for robotic manipulation where human interventions are suboptimal and safety is critical. It introduces SiLRI, a state-wise Lagrangian constrained RL framework that learns a policy by balancing RL and imitation according to per-state human uncertainty using learnable, state-dependent multipliers in a min–max optimization. The approach combines a Q-function, policy, human behavior model, and a Lagrange multiplier network, guiding the policy to imitate reliably in low-uncertainty states while exploiting RL in high-uncertainty states, demonstrating faster convergence to high success rates and robust autonomous performance on a suite of real-world tasks. The results suggest that SiLRI can reduce human labor and improve data efficiency for real-world robotics, with potential for broader applicability in human-in-the-loop RL and constrained optimization settings.

Abstract

Real-world reinforcement learning (RL) offers a promising approach to training precise and dexterous robotic manipulation policies in an online manner, enabling robots to learn from their own experience while gradually reducing human labor. However, prior real-world RL methods often assume that human interventions are optimal across the entire state space, overlooking the fact that even expert operators cannot consistently provide optimal actions in all states or completely avoid mistakes. Indiscriminately mixing intervention data with robot-collected data inherits the sample inefficiency of RL, while purely imitating intervention data can ultimately degrade the final performance achievable by RL. The question of how to leverage potentially suboptimal and noisy human interventions to accelerate learning without being constrained by them thus remains open. To address this challenge, we propose SiLRI, a state-wise Lagrangian reinforcement learning algorithm for real-world robot manipulation tasks. Specifically, we formulate the online manipulation problem as a constrained RL optimization, where the constraint bound at each state is determined by the uncertainty of human interventions. We then introduce a state-wise Lagrange multiplier and solve the problem via a min-max optimization, jointly optimizing the policy and the Lagrange multiplier to reach a saddle point. Built upon a human-as-copilot teleoperation system, our algorithm is evaluated through real-world experiments on diverse manipulation tasks. Experimental results show that SiLRI effectively exploits human suboptimal interventions, reducing the time required to reach a 90% success rate by at least 50% compared with the state-of-the-art RL method HIL-SERL, and achieving a 100% success rate on long-horizon manipulation tasks where other RL methods struggle to succeed. Project website: https://silri-rl.github.io/.

Real-world Reinforcement Learning from Suboptimal Interventions

TL;DR

This paper addresses real-world RL for robotic manipulation where human interventions are suboptimal and safety is critical. It introduces SiLRI, a state-wise Lagrangian constrained RL framework that learns a policy by balancing RL and imitation according to per-state human uncertainty using learnable, state-dependent multipliers in a min–max optimization. The approach combines a Q-function, policy, human behavior model, and a Lagrange multiplier network, guiding the policy to imitate reliably in low-uncertainty states while exploiting RL in high-uncertainty states, demonstrating faster convergence to high success rates and robust autonomous performance on a suite of real-world tasks. The results suggest that SiLRI can reduce human labor and improve data efficiency for real-world robotics, with potential for broader applicability in human-in-the-loop RL and constrained optimization settings.

Abstract

Real-world reinforcement learning (RL) offers a promising approach to training precise and dexterous robotic manipulation policies in an online manner, enabling robots to learn from their own experience while gradually reducing human labor. However, prior real-world RL methods often assume that human interventions are optimal across the entire state space, overlooking the fact that even expert operators cannot consistently provide optimal actions in all states or completely avoid mistakes. Indiscriminately mixing intervention data with robot-collected data inherits the sample inefficiency of RL, while purely imitating intervention data can ultimately degrade the final performance achievable by RL. The question of how to leverage potentially suboptimal and noisy human interventions to accelerate learning without being constrained by them thus remains open. To address this challenge, we propose SiLRI, a state-wise Lagrangian reinforcement learning algorithm for real-world robot manipulation tasks. Specifically, we formulate the online manipulation problem as a constrained RL optimization, where the constraint bound at each state is determined by the uncertainty of human interventions. We then introduce a state-wise Lagrange multiplier and solve the problem via a min-max optimization, jointly optimizing the policy and the Lagrange multiplier to reach a saddle point. Built upon a human-as-copilot teleoperation system, our algorithm is evaluated through real-world experiments on diverse manipulation tasks. Experimental results show that SiLRI effectively exploits human suboptimal interventions, reducing the time required to reach a 90% success rate by at least 50% compared with the state-of-the-art RL method HIL-SERL, and achieving a 100% success rate on long-horizon manipulation tasks where other RL methods struggle to succeed. Project website: https://silri-rl.github.io/.
Paper Structure (23 sections, 2 theorems, 21 equations, 10 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 2 theorems, 21 equations, 10 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Proposed Method: SiLRI
  5. Problem Formulation
  6. State-wise Lagrange Multipliers
  7. Network Architectures and Loss Functions
  8. Human-in-the-loop Training Paradigm
  9. Key Design Choices and Parameters
  10. Experiments
  11. Experimental Setup
  12. Baselines and Metrics
  13. Overall Performance Comparison
  14. Robustness Experiments
  15. Ablation Studies
  16. ...and 8 more sections

Key Result

Lemma 3

For two non-degenerate Gaussians $\mathcal{N}(\bm{m}_1,\Sigma_1)$ and $\mathcal{N}(\bm{m}_2,\Sigma_2)$, the KL divergence admits the closed form

Figures (10)

  • Figure 1: Entropy of Human Interventions Across States. Entropy is estimated with a multivariate normal distribution model. In low-entropy states (blue region), human operators intervene consistently and confidently, whereas in high-entropy states (green region), their interventions are inconsistent, indicating uncertainty.
  • Figure 2: SiLRI Enables Effective Real-world RL from Suboptimal Interventions. Left: A human-as-copilot teleoperation system that enables seamless human intervention. Right: The overall optimization objective of SiLRI. Relaxing the constraint in high-entropy states enables the learned policy $\pi$ to converge to a policy that outperforms human behavior policy $\beta$.
  • Figure 3: Network Components in SiLRI. The networks $Q$, $\pi$, and $\lambda$ are updated asynchronously during data collection using online buffer, while the network $\beta$ is updated periodically after a fixed number of new samples have been added to the intervention buffer.
  • Figure 4: Eight Real-world Manipulation Tasks on Two Embodiments. (A) Pick-Place Bread (B) Pick-up Spoon (C) Fold Rag (D) Open Cabinet (E) Close Trashbin (F) Push-T (G) Hang Chinese Knot (H) Insert USB.
  • Figure 5: Training Curves of Episode Length, Intervention Ratio, and Success Rate. We train four online methods on five different tasks. To ensure consistency, all methods within the same task are operated by the same human operator.
  • ...and 5 more figures

Theorems & Definitions (3)

  • Lemma 3: KL Divergence Between Gaussians
  • Theorem 4: From a KL Constraint to a Mean-Distance Constraint
  • proof