CRL-VLA: Continual Vision-Language-Action Learning

TL;DR

This work tackles continual post-training for Vision-Language-Action models in open-world robotic tasks by deriving a unified stability–plasticity bound that ties performance on old and new tasks to the goal-conditioned advantage magnitude $M_g$ scaled by policy divergence. It proposes CRL-VLA, a dual-critic framework with a frozen Goal-Conditioned Value (GCV) critic and a trainable Monte Carlo (MC) critic, coupled with a Goal-Conditioned Value Formulation to decouple semantic stability from adaptive learning. The training objective combines PPO with KL-based constraints and value-based regularizers to bound $M_{ ext{old}}$ and $M_{ ext{new}}$, enabling effective forward adaptation while mitigating forgetting. Empirical results on LIBERO-derived task suites demonstrate improved anti-forgetting and forward transfer, showing that the approach outperforms strong baselines in both single-task and multi-task continual learning scenarios. These findings advance reliable lifelong reasoning and manipulation for embodied agents operating across diverse, non-stationary instruction-grounded tasks.

Abstract

Lifelong learning is critical for embodied agents in open-world environments, where reinforcement learning fine-tuning has emerged as an important paradigm to enable Vision-Language-Action (VLA) models to master dexterous manipulation through environmental interaction. Thus, Continual Reinforcement Learning (CRL) is a promising pathway for deploying VLA models in lifelong robotic scenarios, yet balancing stability (retaining old skills) and plasticity (learning new ones) remains a formidable challenge for existing methods. We introduce CRL-VLA, a framework for continual post-training of VLA models with rigorous theoretical bounds. We derive a unified performance bound linking the stability-plasticity trade-off to goal-conditioned advantage magnitude, scaled by policy divergence. CRL-VLA resolves this dilemma via asymmetric regulation: constraining advantage magnitudes on prior tasks while enabling controlled growth on new tasks. This is realized through a simple but effective dual-critic architecture with novel Goal-Conditioned Value Formulation (GCVF), where a frozen critic anchors semantic consistency and a trainable estimator drives adaptation. Experiments on the LIBERO benchmark demonstrate that CRL-VLA effectively harmonizes these conflicting objectives, outperforming baselines in both anti-forgetting and forward adaptation.

Figures (4)

• Figure 1: The proposed CRL-VLA method. CRL-VLA is a continuous learning framework for VLA models that treats the goal-conditioned advantage magnitude as a key factor determining the trade-off between stability and plasticity. CRL-VLA achieves this insight through a dual critic architecture and a PPO loss with several regularization terms, enabling asymmetric tuning that efficiently adapts to new tasks while preserving prior knowledge.
• Figure 2: Performance comparison of various algorithms during continual learning. Specific task settings in Appendix \ref{['para:task2']}.
• Figure 3: The impact of different values of the loss weights of $\mathcal{L}^{Q}_{\mathrm{GCV}}$ (left), $\mathcal{L}^{V}_{\mathrm{GCV}}$ (middle), and policy KL (right) on the continuous learning performance of the VLA model. Specific task settings in Appendix \ref{['para:task4']}.
• Figure 4: The impact of different values of the loss weights of $\mathcal{L}^{V}_{\mathrm{mc}}$ (left) and $\mathcal{L}^{Q}_{\mathrm{mc}}$ (middle) on the continuous learning performance of the VLA model, and an ablation comparison with and without GCV (right). Specific task settings in Appendix \ref{['para:task4']}.

Theorems & Definitions (21)

• Definition 4.1
• Theorem 4.1: Unified Stability-Plasticity Bounds
• proof
• Remark 4.1
• Corollary 4.1: Controllability of Stability
• Remark 4.2
• Corollary 4.2: Natural Boundedness of Plasticity
• Remark 4.3
• Definition 1.1: Mixture Occupancy Measures
• Definition 1.2: Advantage Magnitude