Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learning

TL;DR

This work tackles the data inefficiency and robustness challenges of deploying reinforcement learning in real-world robotics by proposing SigEnt-SAC, an off-policy actor-critic method that learns from scratch from a single expert trajectory. The core ideas are a sigmoid-bounded entropy term $\mathcal{H}_{\text{sig}}$ to stabilize Q-value updates and a gated behavior cloning component to provide targeted guidance near expert demonstrations, yielding a stable learning signal without large offline datasets. Across D4RL benchmarks and four real-world robots with single-view perception and sparse rewards, SigEnt-SAC achieves rapid convergence to 100% success, demonstrates resilience to noisy demonstrations, and generalizes across morphologies, while maintaining comparable deployment costs. These results suggest a practical path toward low-cost, data-efficient real-world RL deployment in diverse robotic systems.

Abstract

Deploying reinforcement learning in the real world remains challenging due to sample inefficiency, sparse rewards, and noisy visual observations. Prior work leverages demonstrations and human feedback to improve learning efficiency and robustness. However, offline-to-online methods need large datasets and can be unstable, while VLA-assisted RL relies on large-scale pretraining and fine-tuning. As a result, a low-cost real-world RL method with minimal data requirements has yet to emerge. We introduce \textbf{SigEnt-SAC}, an off-policy actor-critic method that learns from scratch using a single expert trajectory. Our key design is a sigmoid-bounded entropy term that prevents negative-entropy-driven optimization toward out-of-distribution actions and reduces Q-function oscillations. We benchmark SigEnt-SAC on D4RL tasks against representative baselines. Experiments show that SigEnt-SAC substantially alleviates Q-function oscillations and reaches a 100\% success rate faster than prior methods. Finally, we validate SigEnt-SAC on four real-world robotic tasks across multiple embodiments, where agents learn from raw images and sparse rewards; results demonstrate that SigEnt-SAC can learn successful policies with only a small number of real-world interactions, suggesting a low-cost and practical pathway for real-world RL deployment.

Figures (7)

• Figure 1: Illustration of negative-entropy effects in soft Q-learning. We compute $\hat{Q}(s,a)$ assuming a Gaussian policy with $\sigma_\pi=0.1$, and assume sampled actions lie within $1.5\sigma_\pi$ of the mean. Top row: the default entropy term (negative entropy; row 1, col 2) lowers $\hat{Q}(s,a)$ for $a=\pi(s)$ and its neighborhood, making max-$Q$ policy improvement more likely to move toward out-of-distribution (OOD) actions. Bottom row: a sigmoid-bounded entropy maps the entropy contribution to a strictly positive, bounded range, yielding a clearer high-$Q$ region and a more well-defined action set for maximization.
• Figure 2: Policy optimization with a negative entropy term is more likely to produce OOD actions; correspondingly, a smaller target policy entropy leads to a higher OOD action ratio. The OOD criterion is defined in Eq. \ref{['eq:p_gate']} with the threshold set to 0.3. We additionally apply LayerNorm to Cal-QL to mitigate spurious OOD actions caused by network oscillations.
• Figure 3: Real-world task suite specification. The figure shows: (Left) an overview of the tasks; (Middle) the unified policy input, consisting of a single-view grayscale local observation; (Right) per-task randomization settings: in Push Cube, the cube is initialized at random positions; in Wheel, both the obstacle and the soccer ball are randomly placed, and the obstacle is movable/collidable; in Quadruped and Humanoid, obstacles are randomly placed within a predefined region and are also movable/collidable.
• Figure 4: Online learning in the one-shot setting, where agents are provided with only one demonstration and must learn to succeed within 1M steps. Compared to other baselines, SigEnt-SAC achieves a higher success rate and converges faster, allows the agent to explore with low target entropy, and exhibits no performance drop after convergence; in contrast, all other baselines either suffer performance drop after convergence or fail to learn throughout the entire training phase.
• Figure 5: Demonstration and learned policy in the real world. (a) On Ball-to-Goal, the demonstration steadily pushes the ball with the uneven front end, while SigEnt-SAC uses its flatter side and a single lateral sway to drive the ball into the goal more quickly and accurately. (b) Steps required to complete each task for the demonstration and the learned policy. The learned policy reduces steps by an average of 40.9% across four tasks; on ball-to-goal, it achieves an 88.46% reduction (26 $\rightarrow$ 3 steps).