CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansion

TL;DR

CLARE tackles the problem of continual, exemplar-free learning for vision-language-action models in robotics by introducing lightweight, modular adapters injected into selective feedforward layers, coupled with an autonomous routing mechanism that activates adapters based on feature similarity. It further employs a dynamic expansion strategy that grows model capacity only when new tasks exhibit substantial novelty, quantified via per-layer autoencoder discriminators; this maintains stability while preserving the base pre-trained representations. Empirically, CLARE outperforms exemplar-based and other continual learning baselines on the LIBERO benchmark, achieving high task performance with minimal parameter growth per task and near-zero forgetting. The work demonstrates that a combination of adapter-based modularity, autonomous routing, and principled expansion can enable robust, long-term continual learning for VLAs in real-world robotics, and lays groundwork for scaling to larger models.

Abstract

To teach robots complex manipulation tasks, it is now a common practice to fine-tune a pre-trained vision-language-action model (VLA) on task-specific data. However, since this recipe updates existing representations, it is unsuitable for long-term operation in the real world, where robots must continually adapt to new tasks and environments while retaining the knowledge they have already acquired. Existing continual learning methods for robotics commonly require storing previous data (exemplars), struggle with long task sequences, or rely on task identifiers for deployment. To address these limitations, we propose CLARE, a general, parameter-efficient framework for exemplar-free continual learning with VLAs. CLARE introduces lightweight modular adapters into selected feedforward layers and autonomously expands the model only where necessary when learning a new task, guided by layer-wise feature similarity. During deployment, an autoencoder-based routing mechanism dynamically activates the most relevant adapters without requiring task labels. Through extensive experiments on the LIBERO benchmark, we show that CLARE achieves high performance on new tasks without catastrophic forgetting of earlier tasks, significantly outperforming even exemplar-based methods. Code and data are available at https://tum-lsy.github.io/clare.

Figures (5)

• Figure 1: Starting from a pretrained vision-language-action model (VLA), CLARE autonomously and continually expands selected feedforward network (FFN) layers with new lightweight adapters. During inference, the most relevant adapters are selected based on feature similarity, captured by learned autoencoder discriminators. By freezing existing parameters and fine-tuning only the new ones at each stage, we can acquire new task-specific knowledge without catastrophic forgetting of previously learned skills.
• Figure 2: CLARE sequentially adds adapters and discriminators as side branches to selected feedforward network layers of a pretrained VLA. Top: During inference, our routing mechanism activates only the most relevant adapter that is linked to the discriminator with the lowest reconstruction error for the input feature. Bottom: During the dynamic expansion phase, if all $z$-scores exceed a threshold $\gamma$, a new adapter and discriminator are added to the corresponding layer. If at least one $z$-score value is smaller than $\gamma$, we only add a discriminator and link it to the most relevant adapter.
• Figure 3: Architecture of our pretrained diffusion transformer (DiT) base policy. We investigate two variants of observation encoding and generative modeling: DiT-EncDec employs a self-attention transformer encoder and a denoising diffusion objective, while DiT-Dec performs a linear projection of the concatenated input tokens and uses a flow matching objective. The potential locations for inserting CLARE adapters are shown as dashed blocks. Our experiments indicate that adding adapters in the encoder module yields the best performance.
• Figure 4: Success rate curves of CLARE and five baselines on the LIBERO-Long benchmark. The solid lines represent the average success rates across three random seeds, and the shaded regions indicate the standard deviations. The results demonstrate that our method achieves a higher overall success rate and more effectively mitigates catastrophic forgetting during continual learning compared to the baselines, despite ER and LOTUS using previous data.
• Figure 5: Ablation study for the dynamic expansion threshold $\gamma$. Increasing $\gamma$ significantly reduces the number of adapters added to the model but slightly reduces the capability to learn new tasks, as shown by the small decrease in AUC and FWT. In contrast, NBT remains at around zero, indicating that the model does not exhibit catastrophic forgetting.