RESample: A Robust Data Augmentation Framework via Exploratory Sampling for Robotic Manipulation

TL;DR

RESample tackles the OOD generalization problem in robotic manipulation by coupling an offline, calibrated action-critic with an exploratory sampling strategy that converts policy-critic disagreements into explicit OOD recovery trajectories. This creates a targeted augmentation loop that trains Vision-Language-Action models to recover from perturbed or unseen states, improving stability and generalization on both simulated LIBERO benchmarks and real-world tasks. The approach leverages a critic anchored to a lightweight proxy actor and employs three regularizers to balance pessimism, near-policy calibration, and data fidelity, enabling efficient, informative data augmentation. Across multiple backbones and task categories, RESample yields consistent, transferable gains, particularly in long-horizon scenarios, and demonstrates practical impact with notable real-world improvements. Limitations include potential critic estimation bias, motivating future work on online critic updates and curriculum-based exploration to further enhance robustness.

Abstract

Vision-Language-Action models (VLAs) have demonstrated remarkable performance on complex robotic manipulation tasks through imitation learning. However, existing imitation learning datasets contain only successful trajectories and lack failure or recovery data, especially for out-of-distribution (OOD) states where the robot deviates from the main policy due to minor perturbations or errors, leading VLA models to struggle with states deviating from the training distribution. To this end, we propose an automated OOD data augmentation framework named RESample through exploratory sampling. Specifically, we first leverage offline reinforcement learning to obtain an action-value network that accurately identifies sub-optimal actions under the current manipulation policy. We further sample potential OOD states from trajectories via rollout, and design an exploratory sampling mechanism that adaptively incorporates these action proxies into the training dataset to ensure efficiency. Subsequently, our framework explicitly encourages the VLAs to recover from OOD states and enhances their robustness against distributional shifts. We conduct extensive experiments on the LIBERO benchmark as well as real-world robotic manipulation tasks, demonstrating that RESample consistently improves the stability and generalization ability of VLA models.

Figures (7)

• Figure 1: Illustration of OOD challenges in demo-driven imitation learning. Top: Conventional approaches rely solely on teleoperated demonstrations, where policies often encounter bottleneck states and fail when facing OOD conditions. Bottom: Our proposed framework leverages exploratory sampling to augment trajectories with fine-grained OOD proxies, enabling policies to explicitly recover from OOD states and reach the target successfully.
• Figure 2: An Overview of RESample Framework. Starting from expert demonstrations (Source Dataset), RESample introduces an exploratory sampling mechanism that leverages policy-critic disagreement to expose OOD states. These challenging states are integrated into an augmented replay buffer and assembled into recovery trajectories, forming an OOD Recovery Dataset. The process establishes a feedback loop: the policy is refined to handle failures while the critic becomes more accurate through exposure to both successes and failures.
• Figure 3: Real-World Experimental Setup. We conduct real-world experiments on a Galaxea A1 robotic arm equipped with a parallel gripper, a wrist-mounted RealSense D435i camera, and an external RealSense L515 camera for a third-person perspective.
• Figure 4: Sample Comparison. Visualization of action sampled by policy before and after applying our RESample framework. The action generated by the augmented policy demonstrates more robustness, while the updated action-critic significantly expands the safety action boundary with higher Q-values overall.
• Figure 5: Cross-Task Augmentation. The augmented data generated from task 2, 4, 5, 8 can be effectively transferred to other tasks within the same category, leading to an additional performance boost of 5-10% on average.