RICL: Adding In-Context Adaptability to Pre-Trained Vision-Language-Action Models

TL;DR

RICL introduces a post-training pipeline that endows pre-trained vision-language-action (VLA) models with in-context learning (ICL) via retrieval-augmented context (RAG). By priming with a small set of demonstrations and keeping the image encoder fixed, RICL trains only the LLM to leverage retrieved context and an action interpolation mechanism, enabling adaptation to unseen objects and novel motions without gradient updates. On pi0-fast-droid, RICL delivers substantial improvements over baselines, achieving 31.25% complete-task success across tasks and up to 83.75% checkpoint completion, with further gains (61.67%) when task-specific finetuning is applied; it also demonstrates robust language grounding and emergent latent actions. The approach offers a practical, data-efficient interface for rapidly teaching new manipulation tasks and highlights directions for scaling demonstrations and handling more drastic scene changes.

Abstract

Multi-task ``vision-language-action'' (VLA) models have recently demonstrated increasing promise as generalist foundation models for robotics, achieving non-trivial performance out of the box on new tasks in new environments. However, for such models to be truly useful, an end user must have easy means to teach them to improve. For language and vision models, the emergent ability to perform in-context learning (ICL) has proven to be a versatile and highly useful interface to easily teach new tasks with no parameter finetuning. Unfortunately, VLAs pre-trained with imitation learning objectives do not naturally acquire ICL abilities. In this paper, we demonstrate that, with the right finetuning recipe and a small robot demonstration dataset, it is possible to inject in-context adaptability post hoc into such a VLA. After retraining for in-context learning (RICL), our system permits an end user to provide a small number (10-20) of demonstrations for a new task. RICL then fetches the most relevant portions of those demonstrations into the VLA context to exploit ICL, performing the new task and boosting task performance. We apply RICL to inject ICL into the $π_{0}$-FAST VLA, and show that it permits large in-context improvements for a variety of new manipulation tasks with only 20 demonstrations per task, without any parameter updates. When parameter updates on the target task demonstrations is possible, RICL finetuning further boosts performance. We release code and model weights for RICL-$π_{0}$-FAST alongside the paper to enable, for the first time, a simple in-context learning interface for new manipulation tasks. Website: https://ricl-vla.github.io.

Figures (10)

• Figure 1: Qualitative comparison between $\pi_0$-FAST-DROID [L] and RICL-$\pi_0$-FAST-DROID [R], with 20 task specific demonstrations for RAG and ICL, on new tasks, including novel objects, motions, and scenes. Additional comparisons can be found in Figure \ref{['fig:qualitative_results_appendix']} in Appendix \ref{['app:additional_results_regentic_tuning']}.
• Figure 2: Architecture of RICL-VLAs, specifically that of RICL-$\pi_0$-FAST.
• Figure 3: [LEFT] Our Franka DROID setup, annotated. [RIGHT] Franka DROID, including the top camera and right camera, moved to a new scene (kitchen sink).
• Figure 4: Success rates of 10 test rollouts from various methods across various tasks represented by stacked bar plots. The lowest bar (dark blue) in each stacked column represents full task success rate, and other bars are the success rates for reaching earlier waypoints. Gray regions represent the fraction of runs that did not even reach the first waypoint for the task. We note that $\pi_0$ refers to $\pi_0$-FAST-DROID and RICL to RICL-$\pi_0$-FAST-DROID in the plots. We also plot the performance of various methods vs the number of demonstration in the idliplate task on the bottom right.
• Figure 5: Qualitative visualization of the reactivity and robustness of RICL-$\pi_0$-FAST-DROID-finetuned on 20 task-specific demonstrations in a dynamic test rollout. In the above, a human randomly perturbs and displaces the primary object during the test rollout. Additional results can be found in Figure \ref{['fig:qualitative_reliability_results_appendix']} in Appendix \ref{['app:additional_results_regentic_tuning']}.