Reflection-Based Task Adaptation for Self-Improving VLA

TL;DR

The paper tackles in situ adaptation of pre trained Vision Language Action models by introducing Reflective Self Adaptation, a dual pathway framework that learns from both failures and successes without human input. Failure driven Reflective RL uses the VLM to causally analyze failures and synthesize dense rewards, accelerating policy exploration, while Success driven Quality Guided SFT grounds the policy in high quality successes to prevent reward hacking and stabilize learning. Extensive LIBERO benchmark experiments show faster convergence and higher final success rates compared to baselines, with ablations confirming the importance of both pathways and curriculum. The approach demonstrates that in situ reflective reasoning can yield robust, autonomous adaptation for complex manipulation tasks in new environments.

Abstract

Pre-trained Vision-Language-Action (VLA) models represent a major leap towards general-purpose robots, yet efficiently adapting them to novel, specific tasks in-situ remains a significant hurdle. While reinforcement learning (RL) is a promising avenue for such adaptation, the process often suffers from low efficiency, hindering rapid task mastery. We introduce Reflective Self-Adaptation, a framework for rapid, autonomous task adaptation without human intervention. Our framework establishes a self-improving loop where the agent learns from its own experience to enhance both strategy and execution. The core of our framework is a dual-pathway architecture that addresses the full adaptation lifecycle. First, a Failure-Driven Reflective RL pathway enables rapid learning by using the VLM's causal reasoning to automatically synthesize a targeted, dense reward function from failure analysis. This provides a focused learning signal that significantly accelerates policy exploration. However, optimizing such proxy rewards introduces a potential risk of "reward hacking," where the agent masters the reward function but fails the actual task. To counteract this, our second pathway, Success-Driven Quality-Guided SFT, grounds the policy in holistic success. It identifies and selectively imitates high-quality successful trajectories, ensuring the agent remains aligned with the ultimate task goal. This pathway is strengthened by a conditional curriculum mechanism to aid initial exploration. We conduct experiments in challenging manipulation tasks. The results demonstrate that our framework achieves faster convergence and higher final success rates compared to representative baselines. Our work presents a robust solution for creating self-improving agents that can efficiently and reliably adapt to new environments.

Figures (4)

• Figure 1: Our Reflective Self-Adaptation framework. Failure loop (yellow): A VLM reflection dialogue synthesizes dense rewards for PPO updates. Success loop (green): Performs prioritized SFT guided by an intrinsic quality assessment.
• Figure 2: Iterative self-improvement on two manipulation tasks. The initial policy (top) is progressively refined through two stages of reflection. Each stage synthesizes a new, corrective reward component (labeled on the side) to overcome successive failure modes.
• Figure 3: Learning curves on our LIBERO-Adapt task suite.
• Figure 4: Visualization of the agent's exploration space on a LIBERO-Adapt task, projected onto the first two dimensions of the action space.