E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning

TL;DR

The paper tackles low sample efficiency and high labor costs in real-world HiL-RL for robotic manipulation. It introduces E2HiL, an entropy-guided sample selection framework that estimates per-sample influence on policy entropy via a covariance-based measure and prunes uninformative samples. An entropy-bounded objective retains only samples with moderate entropy impact, stabilizing entropy dynamics and improving data efficiency. Experiments across four real-world tasks show superior success rates and reduced human interventions compared with the state-of-the-art baseline, underscoring practical benefits.

Abstract

Human-in-the-loop guidance has emerged as an effective approach for enabling faster convergence in online reinforcement learning (RL) of complex real-world manipulation tasks. However, existing human-in-the-loop RL (HiL-RL) frameworks often suffer from low sample efficiency, requiring substantial human interventions to achieve convergence and thereby leading to high labor costs. To address this, we propose a sample-efficient real-world human-in-the-loop RL framework named \method, which requires fewer human intervention by actively selecting informative samples. Specifically, stable reduction of policy entropy enables improved trade-off between exploration and exploitation with higher sample efficiency. We first build influence functions of different samples on the policy entropy, which is efficiently estimated by the covariance of action probabilities and soft advantages of policies. Then we select samples with moderate values of influence functions, where shortcut samples that induce sharp entropy drops and noisy samples with negligible effect are pruned. Extensive experiments on four real-world manipulation tasks demonstrate that \method achieves a 42.1\% higher success rate while requiring 10.1\% fewer human interventions compared to the state-of-the-art HiL-RL method, validating its effectiveness. The project page providing code, videos, and mathematical formulations can be found at https://e2hil.github.io/.

Figures (6)

• Figure 1: Compared with random uniform sampling in HiL-RL, E2HiL avoids early entropy collapse through entropy-guided sample selection, achieving a better exploration–performance trade-off and improving success rates while reducing human costs.
• Figure 2: Entropy-Guided Sample Selection Framework. Our framework consists of two key components: Sample Entropy Dynamics Estimation, where we characterize the entropy dynamics induced by each sample, and Entropy-Bounded Sample Selection, where we prune shortcut and noisy samples by constraining their influence value within dynamic bounds.
• Figure 3: Real-world robot setup. We employ the Lerobot SO-101 as our real-world reinforcement learning platform.
• Figure 4: Comparison with Baseline.E2HiL significantly outperforms state-of-the-art human-in-the-loop RL approaches by achieving higher success rates and requiring fewer human interventions across four tasks. With entropy-guided sample selection, E2HiL further exhibits more stable entropy dynamics, avoiding premature entropy collapse and ineffective updates.
• Figure 5: Left: The policy entropy derivative and covariance during human-in-the-loop RL training in the Touch-Cube task. Right: Covariance of human intervention samples and policy self-exploration samples.