On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning

TL;DR

This paper introduces TT-VLA, a test-time reinforcement learning framework that enables on-the-fly adaptation of Vision-Language-Action models during deployment. By employing a dense, progress-based reward and a value-free PPO, TT-VLA updates the policy online without retraining or access to training data, preserving pretrained priors while addressing distribution shifts. Theoretical analysis justifies the one-step, reward-focused updates, and empirical results show consistent improvements across multiple VLA backbones in both simulation and real-world robotics. The approach demonstrates practical deployment benefits, enabling self-improvement for VLAs in dynamic environments, with discussions on limitations and future extensions to diffusion-based and multimodal settings.

Abstract

Vision-Language-Action models have recently emerged as a powerful paradigm for general-purpose robot learning, enabling agents to map visual observations and natural-language instructions into executable robotic actions. Though popular, they are primarily trained via supervised fine-tuning or training-time reinforcement learning, requiring explicit fine-tuning phases, human interventions, or controlled data collection. Consequently, existing methods remain unsuitable for challenging simulated- or physical-world deployments, where robots must respond autonomously and flexibly to evolving environments. To address this limitation, we introduce a Test-Time Reinforcement Learning for VLAs (TT-VLA), a framework that enables on-the-fly policy adaptation during inference. TT-VLA formulates a dense reward mechanism that leverages step-by-step task-progress signals to refine action policies during test time while preserving the SFT/RL-trained priors, making it an effective supplement to current VLA models. Empirical results show that our approach enhances overall adaptability, stability, and task success in dynamic, previously unseen scenarios under simulated and real-world settings. We believe TT-VLA offers a principled step toward self-improving, deployment-ready VLAs.

Figures (6)

• Figure 1: TT-VLA supplements to SFT/RL-trained VLAs by continuously adapting policies to environment-derived rewards at test time, improving robustness to distributional shifts without retraining.
• Figure 2: Overview of TT-VLA.(a) Overall pipeline. In TT-VLA, a pretrained VLA policy receives an observation and instruction, executes actions in the environment, and receives dense, progress-based rewards computed by a progress estimator. These rewards are used to update the policy online via a value-free PPO objective, enabling continuous within-episode policy adaptation at test time (see §\ref{['subsec:method']}). (b) Effectiveness. TT-VLA consistently improves the performance of diverse VLA backbones across unseen tasks, demonstrating robust generalization and adaptability under evolving conditions or distributional shifts (see §\ref{['subsec:sim-result']}-\ref{['subsec:real-world']}).
• Figure 3: Real-world setup and evaluation. We evaluate nine real-world pick-and-place tasks covering Execution, Vision, and Semantics generalization, with three tasks per category. The results show that TT-VLA consistently improves performance over baseline VLA models in real-world settings.
• Figure 4: Impact of reward design. The results show that our progress-based reward consistently outperforms the standard GAE across tasks and models.
• Figure 5: Real-world case study illustrates how TT-VLA’s instantaneous reward feedback enables rapid recovery from trajectory errors during deployment.

Theorems & Definitions (6)

• Proposition 1: Vanishing learning signal under progress-difference reward
• proof
• Corollary 1: Negative TD bias when $\gamma<1$
• proof
• Lemma 1: One-step collapse of GAE
• proof