SIME: Enhancing Policy Self-Improvement with Modal-level Exploration

TL;DR

SIME tackles the data efficiency bottleneck of imitation learning by enabling policy self-improvement through modal-level exploration during inference, producing diverse multi-modal interactions without retraining. It introduces a latent-space perturbation and an annealing schedule to encourage exploration early in denoising steps, complemented by inter-demo and intra-demo data selection to maximize learning value. Empirical results across RoboMimic benchmarks and a real-world cup-stacking task show substantial improvements over baselines, including up to +12.3% in state-based and +19.9% in image-based metrics, and a dramatic 117.6% real-world gain, highlighting the practical impact of enhanced data diversity and selective learning. The approach is plug-and-play and scalable, offering a path toward more robust, sample-efficient self-improving robotic policies.

Abstract

Self-improvement requires robotic systems to initially learn from human-provided data and then gradually enhance their capabilities through interaction with the environment. This is similar to how humans improve their skills through continuous practice. However, achieving effective self-improvement is challenging, primarily because robots tend to repeat their existing abilities during interactions, often failing to generate new, valuable data for learning. In this paper, we identify the key to successful self-improvement: modal-level exploration and data selection. By incorporating a modal-level exploration mechanism during policy execution, the robot can produce more diverse and multi-modal interactions. At the same time, we select the most valuable trials and high-quality segments from these interactions for learning. We successfully demonstrate effective robot self-improvement on both simulation benchmarks and real-world experiments. The capability for self-improvement will enable us to develop more robust and high-success-rate robotic control strategies at a lower cost. Our code and experiment scripts are available at https://ericjin2002.github.io/SIME/

Figures (6)

• Figure 1: Overview of SIME. With modal-level exploration, the robot can generate more diverse and multi-modal interaction data. By learning from the most valuable trials and high-quality segments from these interactions, the robot can effectively refine its capabilities through self-improvement.
• Figure 2: Pipeline overview. The robot learns a policy from human demonstrations, explores multi-modal interaction behaviors in the reasoning space, collects and selects valuable trajectories and segments, and refines the policy.
• Figure 3: Diversity analysis. We tested 1,000 different scenarios, conducting 10 trials for each. After independently calculating the success rate for each scenario, we analyzed the distribution of success rates. As shown, modal-level exploration allows the policy to exhibit diverse behaviors.
• Figure 4: Trajectory comparison. By conducting 100 tests with the same initial state, we observe that the diffusion policy’s output lacks diversity. However, introducing modal-level selection significantly increases the diversity of the generated behaviors.
• Figure 5: Multi-round self-improvement results comparing the baseline method and our approach on the Can task, starting with 10 and 20 human demonstrations.